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Abstract 

To  emphasize  the  importance  of  sustainment,  the  DoD  Joint  Requirements 
Oversight  Council  addressed  sustained  Materiel  readiness  and  established  a  mandatory 
Key  Perfonnance  Parameter  (KPP)  for  Materiel  Availability;  it  also  established 
supporting  Key  System  Attributes  (KSAs)  for  Materiel  Reliability  and  Ownership  Cost 
(Chainnan  of  the  Joint  Chiefs  of  Staff  Manual  (CJCSM)  3 170.0 1C,  2007). 

Current  guidance  requires  two  numbers:  a  threshold  value  and  an  objective 
value  (Chairman  of  the  Joint  Chiefs  of  Staff  Manual  (CJCSM)  3 170.0 1C,  2007).  No 
distinction  is  made  between  the  approaches  in  establishing  these  values  for  major 
system  acquisitions,  versus  smaller,  modification- focused  efforts  for  existing  systems. 
The  Joint  Staff  proposed  guidance  to  assist  in  determining  these  values  for  major 
acquisition  programs,  but  the  guidance  has  yet  to  be  tested  on  modification  contracts. 
To  assess  its  applicability,  we  performed  a  case  study  of  a  recent  acquisition  program 
under  consideration  by  Air  Mobility  Command.  We  sought  to  apply  the  principles  put 
forth  in  this  draft  guide  prepared  by  the  Office  of  the  Secretary  of  Defense  in 
Collaboration  with  the  Joint  Staff. 

This  research  seeks  to  assist  the  combat  developer  and  program  manager  to 
develop  an  objective,  standard,  repeatable  method  for  quantifying  the  mandatory 
Materiel  Availability  KPP  and  the  associated  Materiel  Reliability  KSA  values 
established  by  the  Joint  Requirements  Oversight  Council. 
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QUANTIFICATION  OF  MANDATORY  SUSTAINMENT  REQUIREMENTS 

I.  Introduction 

Background 

The  May  2003  version  of  DoD  Directive  (DoDD)  5000.1,  “The  Defense  Acquisition 
System,”  references  program  objectives  for  total  life-cycle  systems  management,  to  include 
sustainment  (DoDD  5000.1,  2003).  To  emphasize  the  importance  of  sustainment,  the  DoD  Joint 
Requirements  Oversight  Council  addressed  sustained  Materiel  readiness  and  established  a 
mandatory  Key  Performance  Parameter  (KPP)  for  Materiel  Availability;  it  also  established 
supporting  Key  System  Attributes  (KSAs)  for  Materiel  Reliability  and  Ownership  Cost 
(Chainnan  of  the  Joint  Chiefs  of  Staff  Manual  (CJCSM)  3 170.0 1C,  2007).  KPPs  are  those 
system  attributes  considered  most  critical  or  essential  for  an  effective  military  capability.  KSAs 
are  those  system  attributes  considered  most  critical  or  essential  for  an  effective  military 
capability  but  not  selected  as  a  KPP.  KSAs  provide  decision  makers  with  an  additional  level  of 
capability  prioritization  below  the  KPP  but  with  senior  sponsor  leadership  control  (generally  4- 
star  level,  Defense  agency  commander,  or  Principal  Staff  Assistant)  (Chairman  of  the  Joint 
Chiefs  of  Staff  Manual  (CJCSM)  3 170.0 1C,  2007).  The  values  used  to  describe  each  KPP  and 
KSA  are  defined  by  a  threshold  value  and  an  objective  value.  The  threshold  value  for  an 
attribute  is  the  minimum  acceptable  value  considered  achievable  within  the  available  cost, 
schedule,  and  technology  at  low-to-moderate  risk.  Performance  below  the  threshold  value  is  not 
operationally  effective  or  suitable.  The  objective  value  for  an  attribute  is  the  desired  operational 
goal  achievable  but  at  higher  risk  in  cost,  schedule,  and  technology.  Performance  above  the 
objective  does  not  justify  additional  expense. 
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Current  guidance  requires  both  threshold  and  objective  value  (Chainnan  of  the  Joint 
Chiefs  of  Staff  Manual  (CJCSM)  3170.01C,  2007).  No  distinction  is  made  between  the 
approaches  in  establishing  these  values  for  major  system  acquisitions,  versus  smaller, 
modification-focused  efforts  for  existing  systems.  In  some  cases  a  range  of  stated  values  may 
be  appropriate,  but  in  other  cases  the  best  approach  may  be  to  specify  the  value  as  a  percentage 
change  or  as  a  function  of  other  variables.  The  Joint  Staff  proposed  guidance  to  assist  in 
detennining  these  values  for  major  acquisition  programs,  but  the  guidance  has  yet  to  be  tested 
on  modification  contracts.  To  assess  its  applicability,  we  performed  a  case  study  of  a  recent 
acquisition  program  under  consideration  by  Air  Mobility  Command.  We  sought  to  apply  the 
principles  put  forth  in  this  draft  guide  prepared  by  the  Office  of  the  Secretary  of  Defense  in 
Collaboration  with  the  Joint  Staff. 

Problem  Statement 

Air  Mobility  Command  (AMC)  lacks  a  standardized  method  to  establish  these 
mandatory  threshold  and  objective  values  for  the  Materiel  Availability  KPP  and  Materiel 
Reliability  KSA  for  modifications  to  existing  weapons  systems.  A  draft  guide  is  proposed  by 
the  Joint  Staff  with  the  purpose  of  assisting  program  decision  makers,  but  its  applicability  to  the 
AMC  problem  is  unclear. 

Research  Objective/Questions 

In  order  to  address  this  issue  the  following  research  question  was  investigated: 

Research  Question:  Is  the  draft  Reliability,  Availability,  Maintainability  and  Cost 
Guide  (RAM-C)  that  the  Office  of  the  Secretary  of  Defense  in  Collaboration  with  the  Joint 
Staff prepared  applicable  for  use  on  modification  program? 

To  answer  this  research  question  we  established  five  investigative  questions: 
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Question  1:  What  portions  of  the  guide  are  applicable  to  this  study  and  what  portions 
do  not  apply?  Initial  review  of  the  RAM-C  guide  appeared  to  provide  the  rationale  behind  the 
development  of  the  sustainment  requirements  for  new  weapon  systems.  This  research  sought  to 
detennine  whether  all  or  portions  of  this  document  would  be  useful  in  developing  sustainment 
requirements  for  modifications  of  existing  weapon  systems.  We  evaluated  this  document  in  a 
comparative  manner  to  determine  program  requirement  differences. 

Question  2:  What  modification  program  would  be  a  viable  candidate  for  the  use  in  this 
study?  Some  of  the  decision  criteria  we  examined  were  modification  size,  timing  and  current 
acquisition  phase.  Another  area  of  interest  we  considered  would  be  the  type  of  support  and 
sustainment  contract,  whether  it  is  a  Performance  Based  Logistics  (PBL)  contract  or  a 
traditional  government  sustainment  contract. 

Question  3:  If  the  guide  is  deemed  applicable  for  use  in  modification  or  upgrade 
programs  what  areas  within  the  guide  requires  changes  and  what  changes  are 
recommended? 

Question  4:  How  should  historical  reliability  and  maintenance  data  information 
systems  be  utilized  to  establish  availability  and  reliability  estimates?  We  attempted  to 
detennine  the  feasibility  of  current  Air  Force  data  collection  systems  such  as  Reliability  and 
Maintainability  Infonnation  System  (REMIS),  the  Supply  Management  Analysis  Reporting 
Tool  (SMART),  and  Global  Combat  Support  System  (GCSS)  as  viable  candidates  for  data 
analysis  and  model  creations. 

Question  5:  Does  the  current  body  of  literature  concerning  Performance  Based 
Logistics  support  our  efforts  to  establish  sustainment  performance  parameters? 
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Research  Focus 


The  Air  Force  Air  Mobility  Command’s  Directorate  of  Plans  and  Programs  approached 
us  with  a  request  for  help  in  quantifying  their  KPPs  for  several  ongoing  major  aircraft 
modifications,  and  will  support  our  research.  Therefore,  we  initially  focused  our  research  on 
recommending  a  standardized  method  for  quantifying  Materiel  readiness  sustainment  metrics 
for  Air  Force  Air  Mobility  Command’s  modification  efforts  but  we  designed  our  approach  and 
methods  to  be  applicable  to  all  major  system  modifications  and  upgrades. 

Methodology 

To  achieve  this  objective,  we  used  an  ongoing  AMC  major  aircraft  modification  as  a  case 
study.  We  also  engaged  in  bi-lateral  discussions  with  program  and  subject  matter  experts,  and 
conducted  a  review  of  applicable  literature  to  develop  a  model  that  incorporates  multiple 
factors  impacting  each  sustainment  metrics.  This  model  was  based  on  the  concepts  and 
principles  put  forth  in  the  draft  guide  developed  by  the  Office  of  the  Secretary  of  Defense  in 
Collaboration  with  the  Joint  Staff. 

The  important  purpose  of  sustainment  metrics  was  to  ensure  that  system  perfonnance  and 
program  cost  were  properly  balanced  leading  to  the  Materiel  capability  developed  being 
operationally  effective,  suitable,  and  affordable  for  the  warfighter.  We  utilized  input  from  the 
program  office,  Headquarters  Air  Mobility  Command,  industry,  and  subject  matter  expert  to 
assist  in  performing  sensitivity  analysis  on  the  contributing  factors  that  affected  overall  system 
readiness  levels  and  effectiveness.  These  factors  included  but  were  not  limited  to  reliability, 
maintainability,  supportability  and  ownership  cost  analysis.  By  evaluating  the  effects  of  trade¬ 
offs  performed  on  these  factors  we  attempted  to  develop  a  repeatable  solution  for  the  readiness 
requirements  mandated  by  the  DoD  Joint  Requirements  Oversight  Council. 
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Assumptions  and  Limitations 

Multiple  reliability  and  maintainability  metrics  may  be  applicable  to  a  given  program. 
For  simplicity,  this  discussion  used  the  metrics  applicable  to  the  Air  Force  Air  Mobility 
Command.  The  combat  developer,  with  technical  support  from  the  program  manager 
(especially  in  the  evaluation  of  existing  technological  capabilities),  evaluated  the  achievability 
of  the  minimum  Materiel  reliability  (reliability  required  by  analyzing  the  ability  of  mature  or 
developing  technologies  to  provide  needed  capabilities).  This  analysis  included  historical 
trending  for  predecessor  systems  and  extrapolation  of  trending  results  to  applicable  new 
replacement  technologies  (Maintenance  Division  Joint  Staff  Logistics  Directorate 
(JS/J4/MXD),  2008).  The  accuracy  of  the  values  set  forth  in  this  research  depended  largely  on 
the  completeness  and  accuracy  of  the  data  infonnation  system  used  in  this  research. 

The  requirements  development  process  concludes  when  all  inputs  are  translated  into 
Materiel  Availability,  Materiel  Reliability,  and  Ownership  Cost  (OC)  with  supporting  rationale 
(Maintenance  Division  Joint  Staff  Logistics  Directorate  (JS/J4/MXD),  2008).  For  the  purposes 
of  this  research  we  only  focused  on  the  Materiel  Availability  KPP  and  Materiel  Reliability 
KSA.  We  did  not  address  costs  during  this  study.  It  is  reasonable  to  argue  that  the  values 
achieved  within  this  research  do  not  necessarily  hold  the  most  optimal  values  possible. 
Optimization  requires  an  exploration  of  all  three  of  these  factors  in  combination.  Also,  because 
the  processes  and  procedures  in  this  study  were  focused  directly  towards  modifications,  the 
results  achieved  in  this  study  may  not  be  applicable  to  new  weapon  systems  acquisitions. 
Implications 

The  purpose  of  this  research  is  to  help  develop  a  better  understanding  of  the  processes 
involved  in  developing  reasonable  and  balanced  requirements.  In  achieving  this  we  hope  to 
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increase  the  ability  of  the  Combat  Developer  and  Program  Manager  to  identify  realistic 
threshold  and  objective  values  for  Materiel  readiness  sustainment.  This  research  seeks  to  assist 
the  combat  developer  and  program  manager  develop  an  objective,  standard,  repeatable  method 
for  quantifying  the  mandatory  Materiel  Availability  KPP  and  the  associated  Materiel 
Reliability  KSA  values  established  by  the  Joint  Requirements  Oversight  Council.  This  will 
help  improve  management  oversight  and  lead  to  more  cost-effective  acquisition  programs. 
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II.  Literature  Review 


Introduction 

Sustainment  concerns  regarding  acquisition  programs  are  not  new.  As  early  as  1971, 
DoDD  5000.1  stated  that  “logistics  support  shall  be  considered  as  a  principal  design  parameter” 
in  acquisition  programs  (DoDD  5000.1,  1971).  The  1982  version  of  the  directive  specifically 
referenced  the  concept  of  sustainability  when  stating,  “improved  readiness  and  sustainability 
are  primary  objectives  of  the  acquisition  process  (DoDD  5000.1,  1982).”  In  1996,  this  same 
philosophy  was  stated  as,  “acquisition  programs  shall  be  managed  to  optimize  total  systems 
perfonnance  and  minimize  the  cost  of  ownership  (DoDD  5000.1,  1996).”  the  total  systems 
approach  is  addressed  in  the  current  version  of  the  directive  and  states  that  “the  PM  shall  be  the 
single  point  of  accountability  for  accomplishing  program  objectives  for  total  life-cycle  systems 
management,  including  sustainment.”  Furthermore,  key  defense  documents  (e.g.,  National 
Defense  Strategy  and  Quadrennial  Review)  stress  life-cycle  issues  as  well  (i.e.,  reduced 
footprint,  reduced  cycle  times,  reduced  ownership  costs).  Thus,  the  focus  of  DoD  acquisition 
strategies  has  evolved  from  an  initial  reliance  on  detailed  military  specifications  and 
performance  specifications  and  to  a  life-cycle  systems  view.  The  Sustainment  KPP  approach  is 
the  next  iteration  in  this  evolution  and  refines  the  process  even  further. 

A  technical  report  sponsored  by  the  Boeing  Aerospace  Company  entitled 
“Maintainability/Reliability  Impact  on  System  Support  Costs”,  (Johnson  et  al,  1973), 
highlighted  the  increasing  emphasis  placed  on  the  reduction  of  total  life  cycle  cost  on  both  new 
and  existing  systems.  In  this  report  data  derived  from  specific  equipment/system  programs  (F- 
111,  F-4,  and  A-7D)  showed  that  design  efforts  to  increase  reliability  and  reduce  maintenance 
requirements  per  failure  can  significantly  reduce  equipment/system  life  cycle  costs.  Therefore, 
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they  offer  a  major  opportunity  for  support  cost  savings,  especially  on  equipment  that  is  mission 
essential.  Consequently,  improved  engineering  design  analysis  techniques,  insight,  and  cost 
consciousness  are  needed  (Johnson  et  al,  1973).  This  report  provided  a  methodology  for 
estimating  life  cycle  cost,  primarily  during  the  operational  phase  and  addressed  quantifiable 
savings  that  could  be  determined  during  early  design. 

In  another  technical  report  entitled  “Category  II  FB-1 1 1A  Reliability  and  Maintainability 
Evaluation”,  (Chamblee  et  al,  1972)  results  were  presented  from  the  Category  II  test  program. 
The  aircraft  demonstrated  a  dismal  1.6-hour  mean  time  between  failures  and  a  1.5-hour  mean 
time  between  aircrew  write-ups.  The  overall  aircraft  reliability  was  significantly  degraded  by 
the  low  reliability  of  flight  controls  and  most  avionics  subsystems.  The  contractor  predicted 
that  23.8  maintenance  man-hours  per  flying  hour  would  be  required,  and  48.0  man-hours  were 
actually  measured.  The  results  from  the  perfonnance  of  the  F-  111  series  of  aircraft 
demonstrated  how  important  the  factors  of  reliability  and  maintainability  analysis  early-on  in 
the  acquisition  of  weapons  systems  can  impact  total  life  cycle  cost. 

In  a  memorandum  dated  August  16,  2004,  the  Deputy  Secretary  of  Defense 
directed  measuring  performance  in  tenns  of  Operational  Availability,  Mission  Reliability,  Cost 
per  Unit  of  Usage,  Logistics  Footprint,  and  Logistics  Response  time.  For  consistency,  this 
memorandum  provided  specific  definitions  of  those  metrics  for  use  across  the  Department. 
Current  guidance  directs  their  use  as  the  standard  set  of  metrics  for  evaluating  overall  Total 
Life  Cycle  Systems  Management  (TLCSM)  Metrics  (Under  Secretary  of  Defense,  2005). 

In  a  memorandum  (Under  Secretary  of  Defense,  2005)  to  the  Secretaries  of  the 
Military  Departments,  recommendations  from  the  Defense  Business  Board  were  presented  to 
the  Deputy  Secretary  of  Defense  that  the  Departments  aggressively  pursue  implementation  of 


8 


Performance-Based  Logistics,  for  all  its  weapons,  new  and  legacy  (Under  Secretary  of  Defense, 
2005).  This  is  relevant  because  PBL  concepts  stress  reliability,  maintainability  and 
supportability  as  the  drivers  of  operational  effectiveness  of  a  system  and  play  a  crucial  role  in 
procurement  decision  making  (Kumar,  2007). 

In  an  interview  with  "Government  Executive"  magazine  May,  2006  Chief  of  Naval 
Operations,  Admiral  Michael  G.  Mullen  stated  “We  have  a  tendency  to  look  at  what  it  takes  to 
get  a  program  out  the  door.  We  don’t  think  too  much  about  what  the  life  cycle  [cost]  is.  It's  'Can 
I  build  it?'  I  would  like  us  all  to  be  mindful  of  what  it  costs  to  operate  whatever  we  are  building 
for  whatever  its  life  is  going  to  be  because  I  have  to  pay  that  bill  every  single  year”  (Chief  of 
Naval  Operations,  Admiral  Mullen,  2006). 

The  specific  genesis  for  the  current  sustainment  focus  is  the  Joint  Requirements 
Oversight  Council  (JROC)  Memorandum  161-06  entitled  “Key  Performance  Parameter  Study 
Recommendations  and  Implementation,”  dated  17  August  2006.  Through  this  memo,  the 
JROC  established  a  mandatory  warfighter  Materiel  Availability  Key  Perfonnance  Parameter 
and  identified  Materiel  Reliability  and  Ownership  Cost  as  related  Key  System  Attributes  for 
new  acquisitions  (Chairman  of  the  Joint  Chiefs  of  Staff  Manual  (CJCSM)  3170.01C,  2007). 

The  following  four  life  cycle  sustainment  outcome  metrics  were  subsequently  established  by 
the  Deputy  Under  Secretary  of  Defense  for  Logistics  and  Materiel  Readiness  (DUSD 
(L&MR)):  Materiel  Availability,  Materiel  Reliability,  Ownership  Cost,  and  Mean  Downtime. 
Furthennore,  CJCSM  3170.01C,  “Operation  of  the  Joint  Capabilities  Integration  and 
Development  System,”  was  modified  and  reissued  on  1  May  2007  to  include  these  sustainment 
metrics. 
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Metrics 


Key  Performance  Parameters  are  those  system  attributes  considered  most  critical  or 
essential  for  an  effective  military  capability.  The  Capabilities  Development  Document  (CDD) 
and  the  Capabilities  Production  Document  (CPD)  must  contain  sufficient  KPPs  to  capture  the 
minimum  operational  effectiveness,  suitability,  and  sustainment  attributes  needed  to  achieve 
the  overall  desired  capabilities  for  the  system  (or  systems  if  the  CDD/CPD  describes  an  system 
of  systems)  during  the  applicable  increment.  Failure  to  meet  a  CDD  or  CPD  KPP  threshold 
may  result  in  a  re-evaluation  or  reassessment  of  the  program  or  a  modification  of  the 
production  increments  (Chainnan  of  the  Joint  Chiefs  of  Staff  Manual  (CJCSM)  3170.01C, 
2007). 

Key  System  Attributes  are  those  system  attributes  considered  most  critical  or  essential 
for  an  effective  military  capability  but  not  selected  as  a  KPP.  KSAs  provide  decision  makers 
with  an  additional  level  of  capability  prioritization  below  the  KPP  but  with  senior  sponsor 
leadership  control  (generally  4-star  level,  Defense  agency  commander,  or  Principal  Staff 
Assistant).  In  the  case  of  the  mandated  Sustainment  KPP  (Materiel  Availability),  the 
supporting  Materiel  Reliability  and  Ownership  Cost  KSAs  require  any  changes  to  be 
documented  in  the  subsequent  update  to  the  acquisition  program  baseline  (Chairman  of  the 
Joint  Chiefs  of  Staff  Manual  (CJCSM)  3 170.0 1C,  2007). 

Materiel  Availability 

Materiel  Availability  is  a  measure  of  the  percentage  of  the  total  inventory  of  a  system 
operationally  capable  (ready  for  tasking)  of  performing  an  assigned  mission  at  a  given  time, 
based  on  Materiel  condition.  This  can  be  expressed  mathematically  as  a  proportion  of  the 
number  of  operational  end  items  to  the  total  population  of  end  items.  Materiel  Availability 
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indicates  the  percentage  of  time  that  a  system  is  operationally  capable  of  perfonning  an 
assigned  mission  and  can  also  be  expressed  as  the  proportion  of  uptime  (operational  time)  to 
total  time  (uptime  +  downtime).  Detennining  the  optimum  value  for  Materiel  Availability 
requires  a  comprehensive  analysis  of  the  system  and  its  planned  use,  including  the  planned 
operating  environment,  operating  tempo,  reliability  alternatives,  maintenance  approaches,  and 
supply  chain  solutions.  Materiel  Availability  is  primarily  detennined  by  system  downtime, 
both  planned  and  unplanned,  requiring  the  early  examination  and  detennination  of  critical 
factors  such  as  the  total  number  of  end  items  to  be  fielded  and  the  major  categories  and  drivers 
of  system  downtime.  The  Materiel  Availability  KPP  must  address  the  total  population  of  end 
items  planned  for  operational  use,  including  those  temporarily  in  a  non-operational  status  once 
placed  into  service  (such  as  for  depot-level  maintenance).  The  total  life-cycle  timeframe,  from 
placement  into  operational  service  through  the  planned  end  of  service  life,  must  be  included 
(Chairman  of  the  Joint  Chiefs  of  Staff  Manual  (CJCSM)  3 170.0 1C,  2007). 

Materiel  Availability  is  a  number  between  0  and  1 00  that  provides  the  average 
percentage  of  time  that  the  entire  population  of  systems  is  materially  capable  for  operational 
use  during  a  specified  period.  Operational  means  in  a  Materiel  condition  such  that  the  end  item 
is  capable  of  perfonning  an  identified  mission.  Materiel  Availability  measures  the  percentage 
of  the  entire  population  that  is  operational  (Chairman  of  the  Joint  Chiefs  of  Staff  Manual 
(CJCSM)  3 170.01C,  2007). 

Materiel  Reliability 

Materiel  Reliability  is  a  measure  of  the  probability  that  the  system  will  perform  without 
failure  over  a  specific  interval.  Reliability  must  be  sufficient  to  support  the  warfighting 
capability  requirements.  Materiel  Reliability  is  generally  expressed  in  tenns  of  a  mean  time 
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between  failures  (MTBF),  and  once  operational  can  be  measured  by  dividing  actual  operating 
hours  by  the  number  of  failures  experienced  during  a  specific  interval.  Reliability  may  initially 
be  expressed  as  a  desired  failure-free  interval  that  can  be  converted  to  MTBF  for  use  as  a  KSA 
(e.g.,  95  percent  probability  of  completing  a  12-hour  mission  free  from  mission-degrading 
failure;  90  percent  probability  of  completing  5  sorties  without  failure).  Specific  criteria  for 
defining  operating  hours  and  failure  criteria  must  be  provided  together  with  the  KSA.  Single¬ 
shot  systems  and  systems  for  which  other  units  of  measure  are  appropriate  must  provide 
supporting  analysis  and  rationale  (Chainnan  of  the  Joint  Chiefs  of  Staff  Manual  (CJCSM) 

3170. 01C,  2007). 

Materiel  Reliability  =  Mean  Time  Between  Failure 

, „  Total  Operating  Hours 

Materiel  Reliability  = - - - - - 

T otal  Number  of  Failures 

Mean  Down  Time 

Mean  Downtime  (MDT)  is  the  average  total  downtime  required  to  restore  an  asset  to  its 
full  operational  capabilities.  MDT  includes  the  time  from  reporting  of  an  asset  being  down  to 
the  asset  being  given  back  to  operations/production  to  operate.  MDT  includes  administrative 
time  of  reporting,  logistics  and  materials  procurement  and  lock-out/tag-out  of  equipment,  etc. 
or  repair  or  preventive  maintenance  (Chainnan  of  the  Joint  Chiefs  of  Staff  Manual  (CJCSM) 
3170.01C,  2007). 

, ,  ^  Total  Down  Time  for  All  Failures 

Mean  Down  Time  (MDT)  = - 

Total  Number  of  Failures 

Reliability,  Availability,  Maintainability  and  Cost  Guide  (RAM-C) 

The  current  guidance  regarding  life  cycle  sustainment  is  relatively  new.  Although 
policy  has  been  established,  there  is  very  little  guidance  published  to  help  program  managers 
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develop  and  quantify  particular  threshold  and  objective  values  for  the  relevant  metrics 
discussed  above.  The  only  document  is  the  RAM-C  guide  (Maintenance  Division  Joint  Staff 
Logistics  Directorate  (JS/J4/MXD),  2008).  The  associated  guidance  notes  that  detennining  the 
optimum  value  for  Materiel  Availability  requires  a  comprehensive  analysis  of  the  system  and 
its  planned  use.  The  guide  provides  a  series  of  questions  to  consider  for  each  metric. 
Furthennore,  discussions  with  Air  Force  headquarters  planners  suggest  that  current  methods  for 
developing  particular  threshold  and  objective  values  for  the  Materiel  availability  KPP  are  ad- 
hoc,  at  best.  The  particular  relationship  between  the  Materiel  Availability  KPP  and  the 
Materiel  Reliability  &  Ownership  Cost  KSAs  isn’t  clear.  Finally,  it  is  often  questionable 
whether  the  values  used  for  these  measures  by  headquarters  planners  are  contractually 
enforceable  or  even  measureable.  Therefore,  holistic  consideration  of  these  metrics  provides 
the  trade-space  to  optimize  their  achievement  and  provide  a  balanced  solution.  It  also  enhances 
the  end-to-end  Materiel  readiness  value  chain  perspective  being  promoted  throughout  the  DoD. 
Performance  Based  Logistics 

While  the  research  literature  on  KPP  constructs  appears  to  be  nonexistent,  significant 
research  has  been  done  in  the  context  of  Performance  Based  Logistics  (PBL)  support  concepts. 
This  is  relevant  because  a  weapon  system’s  key  perfonnance  parameters  should  possess  linkage 
to  the  metrics  used  to  assess  a  contractor’s  PBL  success.  Mahon  (2007)  states  the  stakeholder 
team’s  job  is  to  develop  perfonnance  criteria  and  metrics  that  are  straightforward,  measurable, 
achievable,  and  are  tied  to  requirements  provided  by  the  warfighter.  Metrics  are  typically 
effectiveness  driven,  such  as  assured  system  availability  (percent)  and  assured  component 
availability  (percent);  or  efficiency-driven,  such  as  assured  number  of  flying  hours,  assured 
number  of  sorties,  etc.  Not  all  metrics  will  be  objective  measures.  Some  aspects  of  product 
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support — for  example,  customer  satisfaction  may  be  subjective  measures.  The  top-level  metric 
objectives  established  by  USD  (AT&L)  include: 

•  Operational  Availability  -  percent  of  time  a  system  is  available  for  a  mission  or 
the  ability  to  sustain  operations  tempo. 

•  Operational  Reliability  -  measure  of  a  system  meeting  mission  success  objective; 
e.g.  a  sortie,  tour,  launch,  or  destination  reached  (Mahon,  2007). 

•  Cost  Per  Unit  Usage  -  total  operating  costs  divided  by  the  unit  of  measurement  for 
a  given  system;  e.g.  flight  hour,  launch,  or  miles  driven. 

•  Logistics  Footprint  -  size  or  presence  of  deployed  logistics  support  required  to 
deploy,  sustain,  or  move  a  system. 

Measurable  elements  include  inventory,  equipment,  personnel,  facilities,  transportation 
assets,  and  real  estate.  Mahon  argues  that  the  Air  Force  can’t  justify  PBL  savings  on  the  basis  of 
cost  savings  or  reduced  logistics  footprints,  because  few  PBL  strategies  simultaneously  focus  on 
availability,  reliability,  cost,  and  logistics  metrics.  Some  focus  on  availability,  or  reduced  cost, 
or  reduced  cycle  time  (Mahon,  2007). 

Kumar  (2007)  insists  that  reliability,  maintainability  and  supportability  drive  the 
operational  effectiveness  of  a  system  and  play  a  crucial  role  in  procurement  decision  making. 

Due  to  the  inherent  criticality  of  reliability,  maintainability  and  supportability  in  the  defense 
industry,  several  procurement  strategies  have  been  evolved  to  influence  decision  making  in  the 
design  of  products,  systems,  and  most  recently,  system  of  systems .  PBL  strategies  focus  on 
product  support  for  large-scale,  repairable  systems  such  as  fighter  aircraft  and  similar  weapon 
systems  (Kumar,  2007). 
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Performance  Based  Logistics  uses  reliability,  maintainability  and  supportability  metrics 
to  evaluate  effectiveness  of  a  product.  For  example,  measures  such  as  operational  availability 
(OA)  and  total  cost  of  ownership  (TCO)  are  used  to  evaluate  the  overall  impact  of  the  system 
design.  Apart  from  OA  and  TCO,  measures  such  as  Total  Operation  and  Maintenance  Cost  and 
Logistics  Footprint  are  used  to  evaluate  maintainability  and  supportability  of  the  system 
respectively.  Kumar  states  that  it  is  commonly  perceived  that  perfonnance  based  logistics  is 
likely  to  improve  product  availability,  reliability,  maintainability  and  supportability  at  a  lower 
cost  through  leveraging  commercial  best  practices.  It  is  reported  that  in  the  Iraq  and  Afghanistan 
engagements  the  overall  availability  of  the  F/A-18E/F,  which  has  components  that  incorporate 
PBL,  has  been  85%  compared  to  the  older  F/A-18C/D  which  is  supported  under  the  traditional 
logistics  practices  and  achieved  only  73%  availability.  Thus,  there  is  evidence  that  PBL 
contracts  have  benefited  the  customers  and  has  influenced  the  customers  to  use  PBL  contracts 
instead  of  the  traditional  logistics  practices  under  which  the  customer  either  purchased  or  leased 
all  the  resources  required  to  support  the  system  (Kumar,  2007). 

Simultaneous  optimization  of  reliability,  maintainability  and  supportability  is  a 
challenging  task  since  improving  one  aspect  of  the  design  may  deteriorate  another  aspect. 
Procurement  strategies  such  as  PBL,  used  by  defense  and  public  sector  organizations,  focus  on 
achieving  high  reliability,  maintainability  and  supportability.  Successful  PBL  contracts  require 
suppliers  to  find  the  optimal  mix  of  competing  resources  that  will  simultaneously  achieve 
multiple,  perfonnance  metrics.  Total  cost  of  ownership,  availability,  reliability,  maintainability 
and  supportability  are  all  goals  that  are  considered  simultaneously  when  detennining  the  optimal 
compliment  of  competing  support  resources  (Kumar,  2007). 
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Many  definitions  of  PBL  exist.  In  the  Defense  Acquisition  University’s  2005  publication 
entitled  Performance  Based  Logistics:  A  Program  Manager ’s  Product  Support  Guide,  the 
following  definition  can  be  found: 

Performance  Based  Logistics  (PBL)  is  the  purchase  of  support  as  an  integrated, 
affordable,  performance  package  designed  to  optimize  system  readiness  and  meet 
perfonnance  goals  for  a  weapons  system  through  long-tenn  support  arrangements  with 
clear  lines  of  authority  and  responsibility.  Simply  put,  performance  based  strategies  buy 
outcomes,  not  products  or  services. 

Berko witz  et  ah,  (2005)  developed  this  comprehensive  definition  to  capture  the  essence 
of  this  new  strategy: 

An  integrated  acquisition  and  sustainment  strategy  for  enhancing  weapon  system 
capability  and  readiness  where  the  contractual  mechanisms  will  include  long-term 
relationships  and  appropriately  structured  incentives  with  service  providers,  both  organic 
and  non-organic  to  support  the  end  user’s  (warfighter’s)  objective. 

As  can  be  seen  in  both  of  these  definitions,  long-tenn  relationships  are  integral  to  the 

concept  of  PBL.  While  many  different  types  of  business  relationships  exist,  significant  long- 

tenn  relationships  are  often  referred  to  as  partnerships  (Gardner,  2008). 

The  goal  of  both  acquisition  and  sustainment  is  to  gain  the  most  efficient  and  effective 

perfonnance  for  a  weapon  system  throughout  its  life  cycle.  In  doing  so,  it  is  important  to  realize 

that  acquisition  and  sustainment  are  not  separate  but  simultaneous  and  integrative  issues  that 

require  analysis  and  synthesis  throughout  the  product  life  cycle  (Berkowitz  et  ah,  2005). 

Performance  Based  Acquisition  (PBA)  and  PBL  research  is  mainly  based  on  systems’ 

performance  which  is  necessary  to  provide  mission  capable  assets  for  the  warfighters  to 

accomplish  their  mission.  The  main  purpose  of  the  PBA  and  PBL  is  linking  the  defense 

acquisition  and  support  activities  with  the  warfighters’  needs  in  the  long  term  agreements  with 

the  support  providers,  both  organic  and  non-organic.  Successful  PBL  implementation  provides 

the  same  level  of  support  within  lower  costs  while  diminishing  logistics  footprints. 
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It  can  be  argued  that  the  DoD’s  compelling  reasons  to  partner  with  its  contractors  are  to 
improve  service  to  its  customers,  the  warfighters,  and  to  improve  asset  perfonnance  and  cost 
efficiencies.  By  employing  the  PBL  strategy,  the  DoD  aims  not  only  to  better  meet  the  needs  of 
the  operational  end-users  by  improving  system  perfonnance  and  readiness  (as  indicated  in  the 
aforementioned  definitions  of  PBL),  but  also  to  minimize  the  total  system  life  cycle  costs  and 
logistics  footprints  associated  with  those  systems  (DoDD  5000.1,  2003). 

Reliability  Improvement 

Smith  (2004)  proposes  a  process  for  planning  and  estimating  the  cost  of  a  reliability 
improvement  program  under  a  PBL  construct  that  accommodates  the  effect  of  equipment  aging 
and  associated  reliability  degradation.  The  process  is  a  structured  methodology  utilizing  past 
field  performance  information  as  a  basis  for  predicting  future  reliability  performance.  The 
methodology  makes  use  of  the  predictive  results  as  a  basis  for  estimating  the  level  of  effort 
required  to  satisfy  the  PBL  contract’s  reliability  and  availability  goals.  The  user  will  enter  into  a 
solicitation  for  a  system  reliability  perfonnance  improvement  program  requesting  a  desired 
future  state  that  may  or  may  not  be  based  on  knowledge  of  the  system’s  historical  perfonnance. 
The  supplier  will  negotiate  with  the  user  the  terms  of  the  goal  with  respect  to  the  cost  of 
improving  the  present  demonstrated  reliability  of  the  system  to  the  desired  goal  in  terms  of 
redesigns  and  support.  The  present  demonstrated  field  reliability  of  the  system  must  be  used  as  a 
baseline  and  a  point  of  departure  in  defining  the  level  of  effort  toward  attaining  the  future  goal. 
The  user  may  not  be  satisfied  with  the  present  demonstrated  reliability  and  may  demand 
improvement  through  a  Request  for  Quality  Improvement  term.  The  supplier  then  must  develop 
and  present  his  level  of  effort  estimates  for  improving  the  system  reliability  (Smith,  2004). 
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The  2001  Quadrennial  Review  (QDR)  stated,  “DoD  will  implement  PBL  to  compress  the 
supply  chain  and  improve  readiness  for  major  weapon  systems  and  commodities.”  Research  was 
conducted  at  AF/A-IL’s  request  to  discover  the  best  practices  and  lessons  learned  from  the  AF’s 
progress  in  PBL  implementation.  There  were  five  objectives  of  this  study: 

1 .  Provide  a  common  understanding  of  PBL. 

2.  Evaluate  AF  regulations/guidance  to  determine  they  are  consistent  with  the  intent 
of  existing  DoD  guidance. 

3.  Identify  and  study  PBL  implementation  “best  practices”  within  the  C-17,  F- 1 17, 
JSTARS,  and  selected  programs. 

4.  Identify  and  study  PBL  implementation  “lessons  learned”  within  the  C-17,  F- 117, 
JSTARS,  and  selected  programs. 

5.  Determine  how  well  the  selected  programs  met  the  intent  of  guidance  assessed  in 
objectives  1  &  2. 

Pettingill  and  Knipper  (2004)  and  their  colleagues  found  that  the  F- 1 17,  C-17,  and 
JSTARS  programs  being  studied  are  not  using  high-level  performance  metrics  (e.g.,  mission 
capability  rate,  improved  product  affordability,  system  reliability,  and  logistics  footprint)  that 
measure  their  success  in  meeting  PBL  performance  goals  (Pettingill,  2004). 

According  to  the  Defense  Acquisition  University  Program  Managers  Guide,  PMs  are 
using  metrics  tied  to  the  systems  and  subsystems  managed  by  the  Product  Support  Integrator(s) 
under  contract,  but  not  to  the  weapon  system  as  a  whole,  just  as  pointed  out  by  Pettingill  and 
Knipper.  PBL  success  meeting  lower-level  metrics  for  these  programs  did  not  necessarily 
translate  to  improved  weapon  system  availability,  because  no  direct  correlation  existed  between 
lower-  and  upper-level  metrics.  The  PM  works  with  the  user/warfighter  to  establish  system 
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performance  needs  and  then  works  with  the  product  support  providers  to  fulfill  those  needs 
through  documentation  of  the  requirements  (including  appropriate  metrics)  in  Performance 
Based  Agreements  (PBAs).  An  effective  PBL  implementation  depends  on  metrics  that 
accurately  reflect  the  user’s  needs  and  can  be  an  effective  measure  of  the  support  provider’s 
performance.  Linking  metrics  to  existing  warfighter  measures  of  performance  and  reporting 
systems  is  preferable.  Many  existing  logistics  and  financial  metrics  can  be  related  to  top-level 
warfighter  performance  outcomes.  Although  actual  PBL  strategies,  as  implemented,  may 
delineate  metrics  at  levels  lower  than  the  warfighter  top-level  measures  (e.g.,  system 
availability),  it  is  important  that  the  initial  identification  of  performance  outcomes  be  consistent 
with  the  key  top-level  metric  areas  (Defense  Acquisition  University,  2005). 

Landreth  (2005)  and  his  colleagues  examined  the  PBL  contract  for  a  naval  aircraft 
auxiliary  power  unit,  and  reached  conclusions  similar  to  those  by  Mahon  and  Pettingill  et  al. 
The  contract  meets  best  commercial  practices  by  applying  PBL  at  the  component  level  where 
appropriate  system  performance  data  were  available  to  establish  cost  effective  contract 
arrangements.  The  contract  is  not  a  true  PBL  application  in  that  the  contract  buys  availability 
and  reliability  improvements  at  a  fixed  price  with  required  improvement  schedules.  The 
contract  does  not  provide  positive  incentives  for  the  contractor  to  provide  greater  reliability,  but 
rather  specific  reliability  improvement  deliverables  (Landreth,  2005).  Pecht  and  Thomas 
(2006)  illustrate  the  relationship  between  warranty  and  reliability  (Pecht,  2006),  (Thomas, 
2006).  Thomas  and  Richard  also  propose  a  method  to  establish  reliability  goals,  or  targets  for 
creating  quality  improvement  strategies,  through  reliability  improvements  of  the  components 
(Thomas,  2006). 
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Other  relevant  research  includes  the  work  by  Kang  et  al  (2005)  who  presents  a  suite  of 
spreadsheet  and  discrete-event  simulation  models  that  collectively  estimate  the  value  of 
system-level  responses  to  improvements  in  component  reliability.  Providing  reduced  lifecycle 
cost  and,  at  the  same  time,  improving  operational  availability  are  fundamental  goals  of  the 
Performance-based  Logistics  (PBL)  and  other  logistics  initiatives  of  the  US  Department  of 
Defense.  In  many  PBL  contracts,  the  contractual  arrangements  are  typically  stipulated  at  the 
level  of  individual  components  (such  as  a  fuel  cell)  or  a  logistic  element  (such  as  inventory  of 
certain  spare  parts).  While  achieving  component-level  performance  goals  is  certainly 
important,  what  really  matters  to  a  warfighter  is  the  operational  availability  of  the  weapon 
system.  Hence,  there  is  a  need  to  develop  a  methodology  and  an  apparatus  for  estimating  the 
operational  availability  (Ao)  of  a  weapon  system  based  on  the  component-level  reliability  and 
maintainability  data  (Kang,  2005). 

Some  research  has  focused  attention  on  the  method  of  predicting  requirements  as 
opposed  to  the  metrics  themselves.  According  to  a  study  to  document  improving  the  Materiel 
readiness  of  the  Marine  Corps,  as  requested  by  the  Marine  Corps  Logistics  Command  (MCLC) 
Materiel  Readiness  Integrated  Process  Team  (MRIPT)  Lead,  reliability,  maintainability,  and 
availability  simulation  models  provide  a  time-continuous  reconstruction  of  a  weapons  system’s 
“average”  mission,  which  involves  simultaneous  consideration  of  the  system  and  other  events. 
The  primary  means  of  evaluating  Materiel  readiness  within  the  Marine  Corps  relies  on 
detenninistic  equations.  However,  as  a  system’s  complexity  increases,  so  do  the  number  of 
variables  necessary  to  define  the  system,  and  the  number  of  associated  equations  to  be  solved 
through  detenninistic  methods.  Variables  such  as  in  repair,  re-supply,  partial  degradation,  duty 
factors,  operating  factors,  allowed  downtime  events,  and  all  the  other  complexities  of  real-life 
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system  operations  or  the  complete  list  of  the  potential  Materiel  readiness  drivers  make 
evaluating  Materiel  drivers  circumspect  and  more  difficult  to  write  the  probability  of  success 
formula  required  by  deterministic  methods.  Use  of  a  simulation  model  versus  detenninistic 
equations  would  enable  these  issues  to  be  addressed  (Concurrent  Technologies  Corporation, 
2007). 

Reliability  vs  Maintainability 

Dellert  (2001)  examined  the  impact  of  reliability  and  maintainability  on  the  Operations 
and  Support  (O&S)  costs  and  Operational  Availability  (Ao)  of  the  Comanche  helicopter.  The 
research  focused  on  the  question  of  where  the  Comanche  program  office  should  allocate 
resources  to  minimize  O&S  costs  and  maximize  Ao.  The  research  indicated  that  the  best 
allocation  of  resources  was  to  the  improvement  of  system  reliability.  The  negative  impact  to 
both  O&S  costs  and  Ao  was  significant  if  the  predicted  reliability  goals  were  not  met.  The 
primary  goal  of  this  research  was  to  detennine  the  sensitivity  of  O&S  costs  to  variations  from 
the  predicted  reliability  and  maintainability  values.  The  biggest  concern  in  this  analysis  was  on 
the  impact  of  not  meeting  the  predicted  goals  vice  the  cost  savings  possible  if  the  goals  were 
exceeded.  It  was  determined  that  O&S  costs  were  more  sensitive  to  reliability  levels  below 
predictions  than  maintainability  levels  below  predictions. 

For  each  increment  of  10%  below  predicted  levels,  up  to  40%,  the  O&S  cost  increases 
by  approximately  3%.  Beyond  a  level  of  40%  below  the  predicted  reliability  goals,  the  O&S 
costs  will  begin  to  increase  at  an  exponential  rate.  In  comparison,  failure  to  reach 
maintainability  goals  only  caused  a  1%  increase  in  O&S  costs  for  each  10%  below  predicted 
levels.  This  rate  of  increase  remained  constant  throughout  all  values  of  maintainability  below 
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predicted.  The  impact  on  O&S  costs  from  reliability  values  greater  than  20%  below  predicted 
was  much  more  severe  than  that  experienced  from  similar  maintainability  levels. 

The  greater  impact  of  reliability  on  O&S  costs  was  attributable  to  the  higher  costs 
associated  with  acquiring  repair  and  replacement  parts.  As  the  reliability  decreased  the  number 
of  failures  will  increase.  This  will  result  in  a  substantial  increase  to  the  total  cost  for 
consumables  items.  Conversely,  the  actual  labor  cost  makes  up  a  very  small  amount  of  the 
total  maintenance  cost.  Since  labor  performed  by  military  maintenance  personnel  is  covered 
indirectly  through  their  annual  salary,  the  only  labor  costs  are  for  depot  level  labor  and  any 
contract  labor  that  is  required.  Hence,  decreases  in  maintainability  will  have  a  much  smaller 
affect  on  maintenance  costs  and  O&S  costs  as  a  whole. 

Summary 

The  key  objective  of  this  research  seeks  to  develop  an  objective,  standard,  repeatable 
method  for  quantifying  the  Materiel  Availability  KPP  and  associated  Materiel  Reliability  and 
Ownership  Cost  KSA  values  for  defense  weapon  system  requirements  documents.  We  examined 
possible  solutions  that  would  provide  a  launch  pad  for  program  managers  to  utilize  in  the 
weapon  systems  acquisition  process. 

This  chapter  examined  the  genesis  behind  the  mandatory  sustainment  requirements 
metrics  associated  with  each.  This  chapter  provided  a  review  of  current  literature  concerning 
Performance  Based  Logistics  support  concepts  and  how  these  concepts  have  a  direct  impact  on 
weapons  systems  sustainment  efforts  and  desired  performance  parameters.  We  also  discussed 
some  of  the  literature  pertaining  to  issues  such  as  reliability  improvement,  maintainability  and 
personnel  support.  While  discernable  gaps  exists  on  the  development  and  quantification  of 
specific  KPP  constructs  and  related  KSAs,  the  infonnation  gained  through  review  of  this 
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pertinent  body  of  knowledge  is  very  insightful  of  the  linkage  that  exists  between  PBL  and 
reliability,  availability,  and  maintainability. 
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III.  Methodology 


The  objective  of  this  research  is  to  recommend  a  standardized  method  for  quantifying 
Materiel  readiness  sustainment  metrics  in  military  acquisition  requirements  documents.  To 
achieve  this  objective,  we  utilized  a  case  study  of  a  projected  aircraft  systems  modification, 
discussions  with  subject  matter  experts,  and  a  review  of  the  literature  to  develop  an  appropriate 
decision  model  that  incorporates  multiple  factors  impacting  each  sustainment  metric.  The 
decision  model  included  components  of  predictive  modeling  and  sensitivity  analysis  of  the 
subsystem  components  that  are  incorporated  within  the  projected  upgrade  to  determine  the 
overall  affect  on  Materiel  readiness  values. 

Research  Design 

In  order  to  gain  an  adequate  perspective  on  contract  length  issues  throughout  DoD,  this 
study  included  discussions  with  knowledgeable  personnel  associated  with  a  variety  of 
organizations  and  programs.  Based  on  recommendations  from  the  thesis  sponsor,  Headquarters 
Air  Mobility  Command,  ideas  for  a  currently  ongoing  acquisition  program  were  solicited  for  this 
study.  One  of  the  initial  points  of  contention  was  which  modification  program  would  be  a 
suitable  candidate  for  this  research.  Some  of  the  areas  considered  for  this  program  were  the  age 
of  the  platform.  The  platfonn  must  be  mature  enough  to  provide  the  needed  historical  level  of 
stability  but  also  has  enough  existing  life  span  remaining  to  be  cost  effective.  This  platform  also 
needed  to  be  one  that  was  early  in  its  requirements  development  phase  in  order  to  get  the 
welcomed  support  of  the  program  decision  makers. 

Headquarters  AMC  had  two  major  aircraft  modifications  programs  in  the  early  phases 
of  acquisition.  Both  programs  under  consideration  were  based  on  a  traditional  in-house 
maintenance  and  support  contract.  First,  there  is  the  C-5  Reliability  Enhancement  and  Re- 
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Engining  Program  (RERP).  The  C-5  RERP  was  undertaken  to  remedy  some  deficiencies 
identified  in  the  current  C-5  fleet  and  to  close  the  gap  requirements  for  on-time  airlift  delivery 
of  oversize  and  outsize  cargo.  At  the  heart  of  RERP  acquisition  strategy  is  an  Initial 
Reliability,  Maintainability,  and  Availability  evaluation  of  modified  C-5B  and  C-5A  aircraft. 
The  C-5  RERP  will  achieve  the  required  wartime  75%  MCR  by  integrating  a  new  commercial- 
off-the-shelf  (COTS)  propulsion  system,  upgrading  70  subsystems  and  components  (including 
50  reliability  enhancements),  providing  proper  spares  levels  necessary  for  an  85%  issue 
effectiveness  rate,  and  improving  the  efficiency  of  C-5  phased  inspection  and  maintenance 
programs.  The  C-5M  climb  performance  will  ensure  access  to  preferred  air  traffic  routings 
between  North  America  and  Europe  or  Northeast  Asia,  and  provide  the  capability  to  operate 
with  wartime  planning  factor  loads  from  shorter  runways  on  hot  days.  The  C-5M  engines  will 
meet  worldwide  aircraft  noise  and  pollution  emission  standards.  RERP  does  not  change  the 
communications,  navigation,  and  surveillance  architecture  of  the  C-5  Avionics  Modernization 
Program  (AMP),  and  does  not  communicate  with  external  systems  (Capability  Production 
Document  for  C-5  Reliability  Enhancement  &  Re-Engining  Program  (RERP),  2008). 

The  second  modification  program  underway  for  Headquarters  AMC  is  the 
Communications,  Navigation  and  Surveillance/Air  Traffic  Management  (CNS/ATM)  program. 
The  CNS/ATM  program  is  primarily  a  safety  of  flight  modification.  The  CNS/ATM  program  is 
an  acquisition  effort  to  extend  the  KC-135  as  a  viable  weapon  system  through  fiscal  year  (FY) 
2040.  It  supports  mitigating  capability  gaps  identified  in  the  Initial  Capabilities  Document  for 
Air  Refueling  and  the  Air  Force  Integrated-Capability  Review  and  Risk  Assessment  (I-CRRA), 
anticipated  airspace  restrictions  within  the  global  CNS/ATM  System,  and  overall  KC-135 
shortcomings  in  reliability,  maintainability,  and  supportability.  With  current  capabilities,  the 
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combatant  commanders  lack  sufficient  worldwide  capable  AR  assets  to  accomplish  all  requested 
future  joint  operations.  The  KC-135  CNS/ATM  program  includes  an  integrated  digital  flight 
director  (DFD),  radio  altimeter  (RA),  and  autopilot  (AP)  systems  and  Angle  of  Attack  (AO A) 
(Capability  Development  Document  (CDD)  for  KC-135  CNS/ATM  Program  Version  4.4 
ACAT:  III,  2008). 

Through  careful  consideration  by  Headquarters  AMC  the  KC-135  CNS/ATM  program 
was  deemed  the  best  fit  for  this  study.  This  aircraft  system  upgrade  consists  of  multiple 
subsystem  modifications/component  replacement  and  access  to  a  wide  array  of  historical 
performance  and  maintenance  data  would  be  readily  available.  This  modification  is  still  in  the 
requirements  document  development  phase  which  makes  it  a  viable  platform  for  study.  Also, 
contact  with  subject  matter  experts,  program  personnel,  and  industry  would  be  available  where 
necessary.  Most  importantly,  because  challenges  posed  by  access  to  test  data  on  the  C-5  RERP 
upgrade  components,  AMC  felt  that  the  KC-135  CNS/ATM  program  provided  a  better  scale  in 
both  size  and  complexity  as  a  starting  point. 

Sustainment  Requirements 

As  defined  in  the  Department  of  Defense  Reliability,  Availability,  Maintainability  and 
Cost  Rationale  Report  Handbook,  the  mandatory  KPP  and  two  supporting  KSAs  are: 

•  Materiel  Availability  KPP  -  Measures  the  percentage  of  the  total  inventory  of  a  system 
that  is  operationally  capable  (ready  for  tasking)  of  performing  an  assigned  mission,  at  a 
given  time,  based  on  Material  condition.  Materiel  Availability  also  indicates  the 
percentage  of  time  that  a  system  is  operationally  capable  of  performing  an  assigned 
mission  and  can  be  expressed  as  the  proportion  of  the  number  of  operational  end  items  to 
the  total  population  of  end  items. 
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•  Materiel  Reliability  KSA  -  Measures  the  probability  that  the  system  will  perform  without 
failure  over  a  specified  interval. 

•  Ownership  Cost  KSA  -  Provides  balance  to  the  sustainment  solution  by  ensuring  that  the 
Operations  and  Support  (O&S)  costs  associated  with  Materiel  Readiness  (eg. 
maintenance,  spares,  fuel,  support,  etc.)  are  considered  in  making  program  decisions.  The 
Ownership  Cost  KSA  is  ultimately  based  on  O&S  Cost  Estimating  Structure  elements  as 
specified  in  the  OSD  Cost  Analysis  Improvement  Group  (CAIG)  “Operating  and  Support 
Cost-Estimating  Guide.”  Appropriate  sections  of  this  document  cover  the  specific 
elements  involved  in  cost  estimation  (Maintenance  Division  Joint  Staff  Logistics 
Directorate  (JS/J4/MXD),  2008). 

Sustainment  Requirements  Development 

The  logical  process  of  developing  sustainment  requirements  has  well-defined  activities  to 
arrive  at  values  that  are  realistic,  achievable,  measurable,  documented,  and  therefore  defendable. 
The  activities  are  summarized  below: 

•  The  first  step  in  developing  sustainment  requirements  is  the  preparation  of  a  Draft 
Concept  of  Operations  (CONOPS)  by  the  combat  developer.  The  CONOPS  identifies 
the  role  of  the  system  in  providing  the  capability  needed  by  the  warfighter  in  tenns  of 
how  it  will  be  used  operationally  (Maintenance  Division  Joint  Staff  Logistics 
Directorate  (JS/J4/MXD),  2008). 

•  Following  the  development  of  the  CONOPS,  the  combat  developer  must  articulate  the 
mix  of  ways  the  system  performs  its  operational  role  in  an  Operational  Mode  Summary 
and  Mission  Profile  (OMS/MP).  This  includes  the  relative  frequency  of  the  various 
missions,  which  systems  will  be  involved  in  those  missions,  and  the  types  of 
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environmental  conditions  the  system  will  be  exposed  to  during  the  system  life.  The 
OMS/MP  describes  the  tasks,  events,  durations,  operating  conditions,  and  environment 
of  the  system  for  each  phase  of  a  mission  (Maintenance  Division  Joint  Staff  Logistics 
Directorate  (JS/J4/MXD),  2008). 

•  Following  the  development  of  the  CONOPS  and  OMS/MP  the  combat  developer  must 
decide  what  minimal  operational  tasks  the  system  must  be  able  to  perform  in  order  to 
accomplish  its  mission  and  what  the  associated  mission  essential  functions  are  in  order 
to  identify  and  classify  potential  failures.  This  information  is  documented  in  the  Failure 
Definition  and  Scoring  Criteria  (FD/SC).  The  combat  developer  should  receive 
assistance  in  developing  the  FD/SC  from  the  program  manager  including  sustainment 
and  T&E  activities  (Maintenance  Division  Joint  Staff  Logistics  Directorate 
(JS/J4/MXD),  2008). 

•  The  combat  developer  uses  the  OMS/MP  and  FD/SC  to  conduct  an  analysis  to 
detennine  the  maintenance  and  support  concepts  describing  the  levels  of  maintenance 
and  the  maintenance  activities  that  will  be  conducted  at  each  level.  All  of  this 
information  is  used  to  draft  initial  Materiel  Availability,  Materiel  Reliability,  and 
Ownership  Cost  goals  and  document  supporting  rationale  and  assumptions 
(Maintenance  Division  Joint  Staff  Logistics  Directorate  (JS/J4/MXD),  2008). 

•  The  program  manager  takes  the  above  infonnation  from  the  combat  developer  and 
detennines  what  is  achievable  based  on  technology  maturity  and  other  factors.  The 
combat  developer  and  program  manager  must  enter  into  a  continuous  dialogue  so  that 
appropriate  trade  studies  can  be  completed,  further  analysis  conducted,  and  appropriate 
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trade  decisions  made  (Maintenance  Division  Joint  Staff  Logistics  Directorate 
(JS/J4/MXD),  2008). 

•  Once  the  combat  developer  and  program  manager  have  reached  agreement  on  a 
balanced  solution  with  acceptable  trade-offs  based  on  the  state  of  the  possible,  the 
combat  developer  needs  to  identify  the  appropriate  sustainability  requirements  for 
inclusion  in  the  Capability  Development  Document  (CDD)and  Capability  Production 
Document  (CPD)  (Maintenance  Division  Joint  Staff  Logistics  Directorate 
(JS/J4/MXD),  2008). 

•  In  the  end,  the  sustainment  requirements  must  enable  warfighter  functional 
requirements  and  be  measurable  and  obtainable.  Unrealistic,  missing,  ambiguous, 
and/or  conflicting  requirements  affect  the  development  process,  result  in  unacceptable 
or  unachievable  performance  levels,  and  drive  acquisition  and  sustainment  costs.  All 
requirements  must  carefully  balance  technological  feasibility  with  operational  needs 
and  desires,  and  are  subject  to  trade-off  in  order  to  optimize  Materiel  Availability 
(Maintenance  Division  Joint  Staff  Logistics  Directorate  (JS/J4/MXD),  2008). 

•  The  requirements  development  process  concludes  when  all  inputs  are  translated  into 
Materiel  Availability,  Materiel  Reliability,  and  Ownership  Cost  (OC)  with  supporting 
rationale.  The  resulting  lower  level  requirements,  as  identified  by  the  combat  developer 
and  rationale  are  documented  in  the  CDD  and  the  CPD  depending  on  the  program 
phase.  The  lower  level  requirements,  such  as  Mean  Time  To  Repair,  Administrative 
Delay  Time,  and  Logistics  Delay  Time,  are  used  in  evaluating  the  resulting  Sustainment 
Requirement  values  (Maintenance  Division  Joint  Staff  Logistics  Directorate 
(JS/J4/MXD),  2008). 
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For  the  purposes  of  this  research  we  will  only  focus  of  the  Materiel  Availability  KPP  and 
the  Materiel  Reliability  KSA.  With  this  fact  in  mind  it  will  be  reasonable  to  argue  that  the  values 
achieved  within  this  research  will  not  necessarily  hold  the  most  optimal  values  possible  since  a 
the  most  optimal  value  requires  an  exploration  of  all  three  of  these  factors  in  combination.  We 
will  not  focus  on  the  development  of  a  CONOPS,  OMS/MP,  and  FD/SC  for  this  system  because 
there  will  be  no  differentiation  in  use  between  the  legacy  system  and  the  upgraded  system. 
Maintenance  Concept  and  Support  Plans  Consideration 

The  maintenance  concept  is  a  general  description  of  the  maintenance  tasks  required  in 
support  of  a  given  system  or  equipment  and  the  designation  of  the  maintenance  level  for 
perfonning  each  task.  The  maintenance  concept  is  implemented  through  a  Product  Support 
Plan  (Maintenance  Division  Joint  Staff  Logistics  Directorate  (JS/J4/MXD),  2008). 

Product  Support  is  the  management/technical  activities  and  resources  needed  to 
implement  the  maintenance  concept  and  establish  and  maintain  the  readiness  and  operational 
capability  of  a  weapon  system,  its  subsystems,  and  its  sustainment  infrastructure.  Product 
Support  encompasses  Materiel  management,  distribution,  technical  data  management, 
maintenance,  training,  cataloging,  configuration  management,  engineering  support,  repair  parts 
management,  failure  reporting  and  analyses,  and  independent  logistics  assessments.  While  the 
provider  of  the  support  may  be  Public,  Private,  or  a  Public-Private  Partnership,  the  focus  is  to 
achieve  maximum  weapon  system  availability  at  the  lowest  total  ownership  cost.  Product 
Support  Plans  detail  how  the  sustainment  requirements  and  resources  are  managed  over  the  life 
cycle  (Maintenance  Division  Joint  Staff  Logistics  Directorate  (JS/J4/MXD),  2008).  The 
product  support  plans  for  the  legacy  system  will  be  utilized  for  the  modified  platfonn. 
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Warfighter  Capability  Needs 


Warfighter  needs  are  the  basis  for  development  of  Materiel  systems.  These  needs  are 
usually  framed  by  the  combat  developer  as  a  required  capability  to  perform  a  mission.  For 
example,  a  typical  requirement  for  a  system  might  be  that  it  has  a  “95 -percent  chance  of 
completing  a  12  hour  mission  with  no  mission  affecting  failures.”  The  program  manager 
translates  the  combat  requirements  into  specific  Materiel  Availability,  Materiel  Reliability,  and 
Ownership  Cost  metrics.  The  resulting  metrics  must  fully  define  warfighter  requirements  from 
a  contract  perspective  (Maintenance  Division  Joint  Staff  Logistics  Directorate  (JS/J4/MXD), 
2008). 


Table  1-  Metric  Definitions 


Metric 

Nomenclature 

Definition 

Am 

Materiel  Availability 

Percentage  of  total  systems  available  for  operational  use 

Ao 

Operational 

Availability 

Percentage  of  time  a  system  is  available  for  operational 
use 

Rm 

Materiel  Reliability 

The  probability  that  the  system  will  perform  its  intended 
function  over  a  specified  time  period 

MTBF 

Mean  Time  Between 
Failures 

The  average  time  between  system  failures  under 
specified  conditions 

MTBM 

Mean  Time  Between 
Maintenance 

The  average  time  between  system  maintenance  activities 
under  specified  conditions 

MDT 

Maintenance  Down 
Time 

The  average  down  time  for  maintenance  actions  (includes 
MTTR,  LDT,  and  ADT) 

MTTR 

Mean  Time  To 

Repair 

The  average  time  required  to  repair  the  system  after 
failure 

LDT 

Logistics  Delay  Time 

All  non-administrative  maintenance  delays  involved  in 
repair  actions — including  transportation  of  the  system  to 
the  repair  location,  time  required  to  obtain  necessary 
spares,  time  waiting  for  repair  personnel  availability,  etc. 

ADT 

Administrative  Delay 
Time 

Times  associated  with  processes  not  directly  involved  in 
restoration  or  repair  activities,  such  as  processing  of 
requests,  short  tenn  non-availability  of  repair  facilities,  or 
delays  due  to  establishment  of  higher  priorities. 
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One  important  purpose  of  the  sustainment  metrics  is  to  ensure  that  system  performance 
and  program  cost  are  properly  balanced  leading  to  the  Materiel  capability  developed  being 
operationally  effective,  suitable,  and  affordable  for  the  warfighter. 

The  balanced  solution  will  determine  the  optimal  points  for  reliability  and  sustainment 
cycle  time  early  in  program  development  thus  ensuring  an  acceptable  life  cycle  cost  for  the 
system  consistent  with  needed  mission  functional  perfonnance. 

Supportability  and  maintainability  concepts  considered  should  include  system  Mean 
Down  Time  (MDT)  optimization  and  ease  of  system  maintenance.  MDT  is  reduced  by  limiting 
Logistics  Delay  Time  (LDT)  through  pre-positioning  sufficient  spares  and  an  efficient  supply 
system  ensuring  the  spares  are  available  at  the  right  place  at  the  right  time.  Limiting 
Administrative  Down  Time  (ADT)  is  another  way  to  limit  overall  system  down  time.  ADT  is 
time  required  to  initiate  a  maintenance  action  after  an  issue  surfaces.  Designing  maintainability 
into  the  system  will  reduce  Mean  Time  To  Repair  (MTTR)  again  reducing  MDT  (Maintenance 
Division  Joint  Staff  Logistics  Directorate  (JS/J4/MXD),  2008). 

Materiel  Availability 

Materiel  Availability  (Am)  is  the  sustainment  KPP  for  applicable  systems  as  defined 
previously.  AM  is  a  characteristic  of  the  system’s  design,  support  structure,  and  operational  use 
profile.  When  the  system  capability  is  fully  fielded,  AM  is  defined  by  the  following  equation: 

A  number  of  operational  end  items 
total  number  of  end  items  acquired 

These  point  estimates  are  based  on  the  following  equivalent  definition  of  AM: 
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A  _  (uptime)  _  ( MTBF ) 

A  *i 

(uptime)  +  (downtime)  ( MTBF  +  MDT ) 

Where: 

Uptime  =  Time  the  system  is  available  to  perfonn  designated  mission 
Downtime  =  Total  time  -  Uptime  =  Time  system  is  unavailable  for  tasking 
MDT  =  Total  system  downtime  expected  given  the  anticipated  support  structure 

The  first  step  in  determining  Materiel  Availability  requirements  would  be  to  establish 
the  baseline  availability  and  reliability  of  the  legacy  system  to  be  upgraded.  The  baseline 
metric  for  comparison  will  be  the  actual  MTBF  of  the  CNS/ATM  -replaced  systems  collected 
over  a  12  month  period.  Legacy  baseline  rates  can  be  derived  from  the  Reliability  and 
Maintainability  Infonnation  System  (REMIS)  to  the  subsystem  level. 

Materiel  Reliability 

Materiel  Reliability  is  a  characteristic  of  the  final  system  design  and  is  designated  a 
KSA.  Materiel  Reliability  is  defined  by  the  MTBF  of  the  system.  Key  to  determining  the 
MTBF  for  any  system  or  subsystem  is  to  first  determine  that  system’s  failure  time  distribution. 
For  this  study  the  failure  time  distribution  was  determined  by  taking  the  historical  data 
retrieved  from  a  PRP-4126  report  in  REMIS  and  utilizing  software  such  as  ARENA®  or  JMP® 
to  fit  the  most  applicable  distribution.  This  report  contains  data  representing  actual  failure  and 
repair  historical  data  used  to  detennine  the  mean  times  between  failures  and  mean  time  to 
repair.  These  MTBF  and  MTTR  values  will  be  considered  the  equilibrium  for  all  distributions 
that  are  fitted  in  this  research. 

Data  Collection  and  Analysis 

Determining  which  of  the  maintenance  data  reporting  and  collection  systems  would  be 
appropriate  for  this  study  was  critical  in  establishing  a  baseline  metric  for  analysis.  There  were 
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three  data  sources  under  consideration  for  this  research.  The  Reliability  and  Maintainability 
Infonnation  System  (REMIS)  is  an  on-line  source  of  unclassified  maintenance  and  supply  data 
for  all  USAF  aircraft.  The  maintenance  information  consists  of  reliability  and  maintenance 
factors  at  the  two  through  five  digit  Work  Unit  Code  (WUC)  level.  REMIS  is  the  primary  Air 
Force  data  system  for  collecting,  validating,  editing,  processing,  integrating,  standardizing,  and 
reporting  equipment  maintenance  data,  including  reliability  and  maintainability  data,  on  a 
global,  world-wide  basis.  REMIS  provides  authoritative  infonnation  on  weapon  system 
availability,  reliability  and  maintainability,  capability,  utilization,  and  configuration.  REMIS 
consists  of  an  integrated  database  containing  weapon  system  and  equipment  inventory, 
operational  status,  configuration  management,  Time  Compliance  Technical  Order  (TCTO) 
data,  and  reliability  and  maintainability  analysis  data  (REMIS  Program  Management  Office, 
2008). 

Next,  we  considered  the  Global  Combat  Support  System-Air  Force  (GCSS-AF),  an  Air 
Force  (AF)  family  of  systems  (FoS)  that  is  an  integral  part  of  GCSS,  the  Joint  Combat  Support 
Command  and  Control  FoS.  The  GCSS-AF  mission  is  to  provide  timely,  accurate,  and  trusted 
Agile  Combat  Support  infonnation  to  Joint  and  AF  commanders,  their  staffs  and  ACS 
functional  personnel  at  all  ranks  and  echelons.  In  addition,  GCSS-AF  is  the  means  by  which 
ACS  Automated  Infonnation  Systems  will  be  modernized  and  integrated  to  improve  business 
processes  (Frye,  2004). 

The  Supply  Management  Analysis  Reporting  Tool  (SMART)  provides  users  with  a 
broad  range  of  AF  field  and  depot  supply  chain  visibility,  including  demand  forecasting, 
aircraft  availability,  organic  and  contract  (repair  and  new  buy)  past  delivery,  in  work  and  due- 


34 


in,  requisition  visibility,  etc.  SMART  includes  120  plus  analyses,  designed  to  provide  supply 
chain  personnel  needed  visibility  to  answers  most  questions  (Knight,  2008). 

Discussions  with  the  Logistics  Branch  Requirements  Division,  Headquarters  Air 
Mobility  Command  and  the  135th  Aircraft  Availability  Improvement  Program  Analysis, 
Logistics  Integration  Flight,  550th  Aircraft  Sustainment  Squadron,  827th  Aircraft  Sustainment 
Group  were  undertaken  to  utilize  the  wealth  of  data  analysis  and  program  experience  that  exists 
among  personnel  with  an  extensive  working  knowledge  of  the  KC-135  aircraft  historical 
performance  data.  Through  these  discussions  we  determined  that  REMIS  would  be  the  best 
source  for  the  historical  data  needed  in  this  research.  REMIS  tools  for  more  tailor  configured 
reports  and  request  allowing  less  navigation  of  undesirable  data.  REMIS  also  allows  for  user 
specified  report  formats  such  as  PDF,  Excel,  and  Delimited  Text.  This  provides  for  ease  of 
data  analysis  and  reporting  of  output.  The  REMIS  Program  Management  Office  is  located  at 
Wright-Patterson  Air  Force  Base,  Ohio  allowing  for  face  to  face  consultation  when  necessary. 
Having  direct  access  to  the  REMIS  system  management  personnel  and  close  proximity  to  an 
operational  KC-135  unit  that  utilizes  REMIS  as  to  satisfy  its  analytical  needs  was  extremely 
helpful  considering  the  limited  amount  of  time  available  to  leam  a  complex  data  collection 
system.  Information  pertaining  to  the  reasonable  and  achievable  performance  of  COTS 
equipment  was  obtained  from  industry  through  Headquarters  Air  Mobility  Command. 

Legacy  System  Baseline  Establishment 

The  following  steps  illustrate  how  the  draft  guide  is  used  to  establish  the  baseline 
performance  measures. 
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Step  1:  Determine  the  baseline  reliability  measure 

A  working  relationship  was  established  with  the  121st  Air  Refueling  Wing  (ARW)  for 
the  purpose  of  determining  the  best  tools  and  reports  embedded  within  REMIS  that  would 
benefit  this  research.  The  121st  ARW  is  an  Air  National  Guard  unit  based  near  Columbus, 

Ohio.  They  were  contacted  to  assist  with  this  research  because  of  the  close  proximity  of  their 
location  to  Wright-Patterson  Air  Force  Base,  Ohio.  Also,  the  National  Guard  represents  a 
population  that  brings  a  different  experience  level  and  stability  that  is  not  normally  present  in 
most  active  duty  units  due  to  nonnal  change  in  permanent  duty  cycles  and  movement  to 
different  weapons  systems.  For  this  purpose,  the  121st  was  chosen  as  one  of  two  KC-135 
aircraft  operations  unit  in  this  study.  The  100th  Air  Refueling  Wing  based  out  of  RAF 
Mildenhall  Air  Base,  England  was  chosen  as  the  second  unit  for  study  in  this  research  because 
it  represents  an  active  duty  component  and  an  overseas  unit.  An  overseas  unit  was  chosen 
because  it  presents  a  different  variable  in  possible  LDT  than  was  would  be  present  in  a  stateside 
organization. 

The  MTBF  totals  were  determined  by  extracting  historical  maintenance  data  from 
REMIS.  This  data  encompassed  actual  failure  and  repair  history  of  40  aircraft  over  a  12-month 
time  period.  The  12-month  time  span  in  this  study  is  due  to  a  limitation  within  the  REMIS 
system  that  only  allows  for  only  a  12-month  look-back  from  time  of  request  in  the  PRP  -4126 
On/Off  Equipment  Maintenance  Detail  Reports.  The  PRP -4126  On/Off  Equipment 
Maintenance  Detail  Report  contains  chronological  failure  and  maintenance  data  for  weapons 
systems  that  are  in  the  U.  S.  Air  Force  inventory. 

REMIS  is  accessed  via  the  Air  Force  portal.  Authorization  to  REMIS  is  limited  to  U.S. 
government  use  only  and  performance  of  official  duties  is  a  requirement  for  access.  An 
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evaluation  of  reliability  and  availability  of  these  aircraft  was  done  with  respect  to  possessed 
flight  hours  during  this  operational  range.  Utilizing  statistical  tools  available  in  ARENA®  and 
JMP®  software  and  basic  statistics  embedded  in  Microsoft  Excel®,  the  baseline  aircraft 
reliability  measures  stated  as  MTBF  hours  can  be  established  for  the  entire  aircraft  system 
minus  the  subsystem  to  be  upgraded.  The  same  process  can  be  utilized  to  determine  the  MTBF 
for  each  upgrade  system.  This  allows  for  the  evaluation  of  the  aircraft  as  a  complete  system 
and  allows  for  segregation  of  upgrade  systems  for  later  sensitivity  analysis. 

Once  the  decision  was  made  about  which  reports  would  be  appropriate  for  this  study 
retrieval  of  the  data  was  the  next  step.  Through  coordination  with  the  REMIS  office  at  Wright- 
Patterson  Air  Force  Base,  Ohio  it  was  determined  that  for  ease  of  use,  the  data  would  be 
downloaded  directly  into  Excel®  spreadsheets  for  statistical  analysis  and  charting,  and  further 
analysis  in  ARENA®.  Figure  1  represents  a  sample  of  the  critical  data  used  in  this  study.  One 
of  the  main  requirements  in  determining  time  between  successive  failures  is  the  operating  time 
of  the  equipment  being  evaluated.  With  the  aid  of  the  REMIS  office  and  the  Analysis  Branch 
of  the  121st  ARW  it  was  immediately  evident  that  a  key  field  in  the  PRP-4126  was  not 
reported.  This  field  should  have  contained  current  operating  time  reported  in  flight  hours  for 
each  work  unit  code  failure.  After  discussions  with  the  REMIS  office,  we  determined  that  this 


was  a  known  anomaly  within  the  REMIS  system  that  would  require  at  minimum  12  months  of 
software  rewrite  to  correct. 


Equipmer  Block  Number 

Serial  Numbei  Current  Open  Job  Control  Commanc 

Geograpl  Organization 

Work  Unit  Code 

Transaction  Da  Start  Time  Stop  Time 

Performir 

KC135R 

120 

5700001440 

O.OOE+OO 

81772851  ANG 

NLZL 

0121 ARF  WG 

4199 

15-Jul-08 

1630 

1800 

NLZL 

KC135R 

120 

5700001440 

O.OOE+OO 

81772850  ANG 

NLZL 

0121  ARF  WG 

4199 

9-Jul-08 

1200 

1300 

NLZL 

KC135R 

120 

5700001440 

O.OOE+OO 

81772852  ANG 

NLZL 

0121  ARF  WG 

4210 

17-Jul-08 

1500 

1800 

NLZL 

KC135R 

120 

5700001440 

O.OOE+OO 

81912855  ANG 

NLZL 

0121  ARF  WG 

14AAE 

10-Jul-08 

1200 

1400 

NLZL 

KC135R 

120 

5700001440 

O.OOE+OO 

81912856  ANG 

NLZL 

0121  ARF  WG 

14BD0 

ll-Jul-08 

1800 

2200 

NLZL 

KC135R 

120 

5700001440 

O.OOE+OO 

81772853  ANG 

NLZL 

0121  ARF  WG 

14BEG 

3-JUI-08 

1400 

1800 

NLZL 

KC135R 

120 

5700001440 

O.OOE+OO 

81912850  ANG 

NLZL 

0121  ARF  WG 

14CBA 

9-JUI-08 

1200 

1300 

NLZL 

Figure  1-  PRP-4126  On/Off  Equipment  Maintenance  Detail  Report 
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Further  engagement  with  the  121st  ARW  and  Headquarters  Air  Mobility  Command  led 
to  a  solution  that  was  determined  to  be  feasible  and  a  reasonable  work-around.  This  work¬ 
around  involved  the  basic  assumption  that  as  a  failure  occurred  and  post-flight  write-ups  were 
entered,  maintenance  actions  proceeded  immediately.  First,  we  noted  that  a  start  maintenance 
action  time  is  recorded  when  a  maintenance  crew  begins  work  to  correct  a  write-up  and  a  stop 
time  is  recorded  when  all  work  has  complete  that  results  in  a  return  to  operational  available 
status.  After  sorting  all  data  by  aircraft  serial  number,  start  date,  and  then  start  time,  the  time 
between  successive  failures  for  each  subsystem  was  detennined  using  the  stop  time  of  the 
previous  failure  to  the  start  time  of  the  next  failure.  A  twenty  four  hour  clock  was  used  to 
detennine  this  time  between  failures.  The  assumption  was  that  these  aircraft  would  be 
available  for  operational  use  during  times  between  failures.  Data  received  from  Headquarters 
Air  Mobility  Command  in  coordination  with  industry  was  utilized  to  validate  the  measures 
obtained  from  this  work-around.  We  determined  that  this  method  provided  a  level  of  accuracy 
that  was  consistent  with  the  numbers  that  were  provided  by  Headquarters  Air  Mobility 
Command. 

All  subsystems  identified  for  modification  in  KC-135  CNS/ATM  program  are  required 
for  flight  in-accordance  with  the  Minimum  Essential  Systems  List  (MESL)  for  MDS  KC135  as 
of  December  01,  2008.  A  critical  failure  reported  on  any  one  of  these  system  causing  that 
system  to  be  non-mission  capable  causes  a  non-mission  capable  status  for  the  aircraft  system  as 
a  whole.  Also,  independence  between  the  subsystems  is  exists.  That  is  a  failure  in  one 
subsystem  has  no  impact  on  another  subsystem.  The  series  relationship  between  the  Digital 
Flight  Director  (DFD),  Radio  Altimeter  (RA),  Auto  Pilot  (AP),  and  Angle  of  Attack  (AO A)  is 
represented  by  the  reliability  block  diagram  in  Figure  2. 
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Figure  2-  Subsystem  Reliability  Block  Diagram 

Step  2:  Associate  average  times  to  repair 

The  maintainability  metrics  will  be  detennined  in  the  same  manner  as  the  MTBF. 

These  metrics  will  include  MTTR,  ADT,  and  LDT.  Where  ADT  and  LDT  historical  measures 
are  not  specified  as  a  segregated  value  in  REMIS,  they  are  incorporated  in  the  repair  time; 
hence,  MTTR  times  are  MDT. 

The  Headquarters  AMC  has  required  that  the  KC-135  CNS/ATM  system  shall  not 
shorten  the  interval  for  scheduled  depot  maintenance  for  the  aircraft.  Also,  The  KC-135 
CNS/ATM  system  shall  not  increase  the  KC-135  scheduled  maintenance  downtime.  Since 
upgrade  requirements  pose  no  impact  to  system  overhaul  interval  scheduled  and  scheduled 
maintenance  downtime,  it  will  not  be  required  to  detennine  sustainment  requirements  for  this 
upgrade  program. 

ADT  measures  the  administrative  delays  in  initiating  maintenance.  Examples  of  ADT 
related  delays  are  those  required  to  initiate  a  request  for  repair,  process  paperwork  related  to  the 
repair,  or  approve  the  repair.  LDT  measures  logistics  delays  related  to  repairs.  Examples  of 
LDT  delays  are  delays  in  spares  availability,  maintenance  personnel  shortages,  transportation 
delays  (to/from  maintenance  locations),  etc  (Maintenance  Division  Joint  Staff  Logistics 
Directorate  (JS/J4/MXD),  2008). 

Step  3:  Calculate  the  resulting  Baseline  Materiel  Availability 

A  _  (uptime)  _  {MTBF) 

M  (uptime)  +  (downtime)  {MTBF  +  MDT) 
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r  _  MTBF 
r  +  A~  MTBF+MDT 


where  r  =  MTTR,  and  X  =  MTBF 
Step  4:  Determine  Spares  Requirement 

Spares  may  include  the  number  of  spare  systems  as  well  as  the  number  of  removable 
components  and  parts  spares.  Establishing  adequate  spares  support  can  have  as  much  impact 
on  system  availability  as  the  inherent  reliability  and  maintainability. 

Sensitivity  Analysis 

In  addition  to  recommendations  brought  forth  by  the  draft  guide,  sensitivity  analysis 
was  performed  on  the  recommended  values  established  by  industry  to  ensure  first,  that  the 
values  represent  achievable  measures.  Also,  through  sensitivity  analysis,  program  officials  can 
see  the  impact  that  some  variables  established  from  historical  performance  achievements  may 
have  on  reliability  and  availability.  This  evaluation  provides  the  trade  space  in  which  decisions 
can  be  made  about  optimal  reliability  improvement  measures  versus  costs. 
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IV.  Data  Analysis 


The  data  retrieved  from  REMIS  represents  actual  failure  and  repair  history  for  the  two 
units  of  study  over  a  12  month  flying  period.  This  data  was  analyzed  utilizing  the  ARENA 
Input  Analyzer  “fit  all”  tool  to  determine  the  MTBFs  for  each  legacy  subsystem  that  is  to  be 
upgraded  in  addition  to  the  rest  of  the  systems  that  make  of  the  aircraft  minus  the  four  upgrade 
systems.  For  the  DFD,  RA,  AP,  and  AOA  subsystems,  while  the  p-values  reject  the  null 
hypothesis  that  the  data  sampled  is  that  of  a  Weibull  distribution,  information  presented  by 
Banks  et  al  (2005)  suggest  that  a  large  sample  size,  such  as  those  present  in  this  study  may 
causes  a  rejection  of  all  candidate  distributions.  Because  of  this  the  associated  histograms  in 
appendix  3  were  utilized  to  present  evidence  that  the  Weibull  distribution  does  provide  an 
appropriate  fit.  The  same  holds  true  for  the  remaining  aircraft  systems  minus  the  upgrade 
systems.  This  data  is  of  an  Exponential  distribution.  The  MTBF  values  and  failure 
distributions  are  shown  in  Table  2. 

Determine  Baseline  Reliability  and  Availability 


Table  2-  Failure  Time  Distribution  Data 


Subsystem 

Distribution 

MTBF 

(Fit  Hrs) 

Kolmogorov 

Smirnov 

p-value 

Sample 

Size 

Aircraft 

Exponential 

12 

<0.01 

5082 

DFD 

Weibull 

717 

<0.01 

154 

RA 

Weibull 

370 

<0.01 

134 

AP 

Weibull 

417 

<0.01 

352 

AOA 

Weibull 

398 

<0.1 

49 
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Except  for  the  AOA  subsystem,  we  used  histograms  to  determine  the  appropriate  repair 
time  distributions.  The  MTTR  for  each  subsystem  is  represented  by  the  data  in  Table  3.  The 
associated  histograms  are  shown  in  Appendix  4. 


Table  3-  Repair  Time  Distribution  Data 


Subsystem 

Distribution 

MTTR 

(Hours) 

Kolmogorov 

Smirnov 

p-value 

Sample 

Size 

Aircraft 

Beta 

1.08 

<0.01 

5083 

DFD 

Fognormal 

1.83 

<0.01 

155 

RA 

Fognormal 

1.56 

<0.01 

135 

AP 

Beta 

1.93 

<0.01 

353 

AOA 

Exponential 

2.28 

>0.15 

50 

The  next  step  in  determining  the  system  baseline  availability  is  to  calculate  the  MTBF 
for  the  system. 

In  general,  for  the  fitted  failure  rate  values  obtained  from  Input  Analyzer  in  ARENA® 

1 


MTBF ' 


System 


where  MTBF;  =  mean  time  to  failure  of  the  z'th  component/subsystem. 
The  system  MTBF  is  therefore  given  by: 


MTBF =  1  /  [(— )  +  (—)  +  (— )  +  (—)  +  (— )]  =  10.83  Hours 
12  717  370  417  398 


System 


(4.1) 
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The  system  MTBF  for  just  the  subsystems  that  are  to  be  upgraded  would  be 
extrapolated  and  calculated  using  the  same  method.  The  subsystems  to  be  upgraded  system 
MTBF  value  was  determined  to  be: 


MTBF,,__ =  1  /  [(— )  +  (— )  +  (— )  +  (— )]  -  1 1 1 .0 1  hours 

717  370  417  398 


Upgrade- System 


(4.2) 


Next,  determine  the  system  MTTR.  This  is  accomplished  as  follows: 


1 


1 


1 


1 


1 


MTTRSystem  =  [(— )  +  (— )  +  (— )  +  (— )  +  (—)]  =  3  •  Hours 


1.08  1.83  1.56  1.93  2.28 


(4.3) 


Resulting  Materiel  Availability: 


Am  = 


(uptime) 


(. MTBF ) 


(uptime)  +  (downtime)  ( MTBF  +  MDT ) 


A„  =■ 


MTBF 


r  +  A  MTBF  +  MDT 


where  r  =  l/MTTRsystem  and  X  =  l/MTBFsystem. 

The  Baseline  Materiel  Availability  for  the  12  month  historical  data  is  therefore: 


A„  =■ 


.3257 


M 


.3257 +  .0923 


■  =  .7791 


(4.4) 


To  ensure  the  validity  of  this  process,  performance  measures  output  by  the  135th 
Aircraft  Availability  Improvement  Program  Analysis,  Logistics  Integration  Flight,  550th 
Aircraft  Sustainment  Squadron,  827th  Aircraft  Sustainment  Group  were  cross-referenced.  The 
values  they  reported  for  the  specific  time  period  of  this  research  was  an  approximate  match  to 
our  measures. 
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Resulting  Minimum  Number  of  Spares  Required 

Assuming  instantaneous  replacement  of  failed  subsystem  component  with  spares  and 
that  all  failed  subsystem  components  are  repairable  (in  order  to  simplify  the  example),  the 
minimum  number  of  spares  required  can  be  determined  by: 


Am  =  percentage  of  aircraft  operational  = 


40  arcft 
X 


0.7791 


Solving  for  X: 


(4.5) 


X  = 


At)  arcft 
0.7791 


=  52  after  rounding  up 


Number  of  spares  =  52  -  40.  To  keep  40  aircraft  operational  on  average  for  1  year,  12 
spares  are  required. 

Reliability  Sensitivity  Analysis 

The  first  step  was  to  evaluate  the  sensitivity  of  the  model  to  changes  in  multiple 
variables  that  play  a  factor  in  overall  reliability  and  availability.  For  this  research  we  only 
evaluated  the  effect  of  changing  one  variable  at  a  time,  not  focusing  on  interactions  between 
variables.  A  key  variable  to  consider  when  determining  how  changes  to  individual  and  system 
reliability  may  be  affected,  would  be  to  perfonn  sensitivity  analysis  on  the  individual 
subsystem  scale  parameters  (9).  This  will  enable  the  developer  to  determine  what  impact  that 
different  characteristic  life  values  will  have  on  the  reliability  of  the  subsystem/components  and 
the  system  reliability.  Each  of  the  subsystems  to  be  upgraded  in  this  study  was  determined  to 
have  a  Weibull  failure  distribution.  We  performed  this  evaluation  using  the  shape  parameters 
(P)  established  from  failure  data  obtained  in  the  baseline  measures. 

We  began  by  establishing  values  based  on  the  recommended  objective  values  provided 
by  industry’s  COTS  measures.  This  would  enable  us  to  determine  the  probability  of  meeting 


44 


these  objective  measures.  Table  4  shows  industry  recommended  threshold  and  objective 
values. 


Table  4-  Industry  Threshold  and  Objective  Values 


THRESHOLD 

OBJECTIVE 

Beta 

DFD 

1545  hrs 

2705  hrs 

43.4 

RA 

773  hrs 

1352  hrs 

14.3 

AP 

657  hrs 

966  hrs 

35.3 

AOA 

927  hrs 

1159  hrs 

6.69 

We  began  our  analysis  by  using  a  value  that  is  1  percent  higher  than  industry’s 
recommended  objective  value  and  increased  that  measure  incrementally  by  1  percent  through 
95  percent  above  that  recommended  objective  value  to  detennine  the  influence  that  it  has  on 
system  reliability.  Except  for  the  AO  A  threshold,  all  values  at  1  percent  and  above  objective 
have  at  least  a  100  percent  probability  of  meeting  threshold  levels  so  we  will  only  evaluate 
achieving  the  objective  values.  As  shown  in  the  calculations  below  the  one  percent  above 
objective  value  achieves  an  81  percent  probability  of  meeting  the  AOA  threshold  value. 

R=e-(/f 


where  t  =  MTBF Threshold  and  6  =  one  percent  above  MTBF objective- 


^=<-(92Xl71)6"=-8111 


where  t  =  MTBFobjective  and  6  =  one  percent  above  MTBF  objective- 


R 


DFD 


=  .5214 


One  percent  incremental  increases  above  recommended  objective  values  for  each 
subsystem  are  shown  in  Table  5. 


Table  5-  Subsystem  Reliability  at  Different  percentages  Above  MTBF  Objective  Values 
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%  Above 

.01 

.02 

.03 

.04 

.05 

.06 

.07 

.08 

DFD 

.5214 

.6544 

.7574 

.8329 

.8862 

.9230 

.9481 

.9650 

RA 

.4218 

.4706 

.5208 

.5649 

.6091 

.6472 

.6847 

.7166 

AP 

.4990 

.6048 

.7033 

.7809 

.8348 

.8801 

.9134 

.9355 

AOA 

.3932 

.4161 

.4406 

.4627 

.4861 

.5089 

.5292 

.5506 

Since  system  reliability  is  a  probability,  the  system  reliability  may  be  determined  from 
subsystem/component  reliabilities.  Components  within  a  system  may  be  related  or  configured 
to  one  another  in  two  primary  ways:  in  either  a  serial  or  a  parallel  configuration.  In  series  all 
components  must  function  for  the  system  to  function.  In  a  parallel,  or  redundant,  configuration, 
at  least  one  component  must  function  for  the  system  to  function.  All  components  in  this  system 
are  considered  critical  for  operation.  Under  this  concept  a  failure  in  any  one  component  or 
subsystem  would  render  the  aircraft  inoperable.  All  components  must  be  functional  for  the 
system  to  be  operational.  This  series  relationship  is  represented  by  the  reliability  block 
diagram  of  figure  3. 


Figure  3-  System  Reliability  Block  Diagram 

The  Upgrade  System  Reliability  is  therefore: 


R, 


'Upgrade  System 


where  i  =  the  /th  subsystem/component. 

The  System  Reliability  at  one  percent  above  the  objective  value  is  therefore: 


RUpgrade  System  =  -52 14  x  .42 1 8  x  .4990  X  .3932  =  .0432 
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Table  6  shows  the  upgrade  system  reliability  increases  at  one  percents  incremental 
increases  above  the  recommended  objective  values. 

Table  6  -  Upgrade  System  Reliability  Increase  versus  Scale  Parameter  (0)  Increase 


0  Increase 

.01 

.02 

.03 

.04 

.05 

.06 

.07 

.08 

R  System 

.0432 

.0775 

.1222 

.1700 

.2190 

.2676 

.3138 

.3562 

In  establishing  the  system  level  reliability  requirements,  both  system  performance  and 
the  cost  associated  with  system  performance  must  be  considered.  The  trade  space  for  the 
sustainment  requirements  is  detennined  by  the  threshold  and  objective  values  determined  for 
Materiel  Availability,  Materiel  Reliability,  and  Ownership  Cost.  Internal  trade-offs  are  made 
to  develop  the  optimal  system  for  the  given  acquisition/sustainment  approach  (for  example, 
increasing  or  decreasing  the  Materiel  Reliability  values  to  reduce  the  overall  LCC).  Even 
though  cost  analysis  will  not  be  thoroughly  explored  during  this  study,  figures  4,  5,  6,  7,  and  8 
clearly  demonstrate  the  trade  space  that  exists  between  reliability  improvement  and 
characteristic  life  that  is  associated  with  reliability  improvements.  For  this  research  we  are 
only  considering  the  impact  of  the  factors  that  affect  achieving  quantifiable  and  reproducible 
sustainment  metrics. 
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Figure  4-  DFD  Reliability  vs  0  Increase 


Figure  5-  RA  Reliability  vs  0  Increase 


48 


Figure  6-  AP  Reliability  vs  0  Increase 


Figure  7-  AOA  Reliability  vs  0  Increase 
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Figure  8-  System  Reliability  vs  0  Increase 

The  sensitivity  analysis  performed  demonstrates  the  ability  of  the  procedures  utilized  in 

the  study  to  provide  decision  makers  with  additional  tools  that  provide  visibility  of  the  trade¬ 
offs  that  may  be  capitalized  upon.  For  example,  the  rate  of  return  on  additional  investment  to 
improve  the  reliability  of  the  DFD,  RA,  and  AP  would  provide  a  much  better  dollar  for  dollar 
return  than  would  be  realized  if  additional  funds  were  allocated  toward  improving  the 
reliability  of  the  AOA  subsystem.  The  concepts  in  this  guide  would  allow  developers  to  meet 
the  warfighters  requirements  in  a  realistic  manner  while  simultaneously  minimizing  costs. 
Materiel  Availability  Sensitivity  Analysis 

Once  analyses  pertaining  to  reliability  measures  were  completed,  the  next  sustainment 
measure  of  interest  was  availability.  Just  as  accomplished  with  the  reliability  measures,  we 
first  evaluated  the  impact  that  the  industry  recommended  values  had  on  availability.  The 
system  threshold  and  objective  MTBF  values  of  220  hours  and  333  hours  respectively,  were 
evaluated  to  determine  what  noticeable  impact  is  recorded  from  the  baseline  availability 
measures  compare  to  the  values  recorded  from  equation  4.4.  Following  this,  we  evaluated 
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availability  at  10  percent  incremental  increases  above  the  recommended  objective  value.  The 
impact  on  overall  availability  was  examined  from  the  10  percent  above  the  recommended  value 
continuing  through  a  500  percent  increase. 

Availability  at  industry  recommended  threshold  values  would  be  determined  by: 

,  r  .  r 

A  = _ * _ 

v  4-  A  V  +  A 

System  -  upgrade  Subsystem  upgrade 


The  Threshold  Materiel  Availability  is  therefore: 


A,,  =- 


.3257 


.3257 


.3257 +  .0833  .3257 +  .0046 


=  .7852 


where  r  1/MTTRsystem  and  2 system -upgrade  1  /MTBF System  -  upgrade  ( 1 1 2  hours)  and 
^ Subsystem  upgrade  1/MTBF  Subsystem  upgrade  (1/220  hours). 


(4.6) 


Availability  at  industry  recommended  objective  values  would  be; 


=■ 


.3257 


.3257 


.3257 +  .0833  .3257 +  .0030 


=  .7890 


(4.7) 


where  .0030  =  1/333  hours. 

The  subsystems  to  upgraded  baseline  system  MTBF  value  was  established  as  1 1 1  hours. 
The  values  of  220  hours  and  333  hours  represent  an  increase  of  98  percent  and  200  percent.  As 
seen  from  the  calculated  availability  values,  the  increase  in  availability  for  these  proposed 
reliability  improvements  equate  to  only  a  small  amount.  The  impact  of  the  10  percent 
incremental  increases  above  the  recommended  objective  value  is  represented  in  figure  9. 
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Figure  9-  Objective  Availability  vs  Upgrade  MTBF  Increase 
Repair  Time  Sensitivity  Analysis 

Finally,  there  may  be  limits  to  what  may  be  achieved  through  reliability  improvements. 
Another  area  to  consider  when  examining  ways  to  reduce  Life  Cycle  Costs  is  to  explore  the 
maintainability  of  the  system.  This  process  begins  by  defining  maintainability  goals.  The 
detennination  of  these  goals  coincides  with  the  reliability  specifications.  Trade-offs  between 
reliability  and  maintainability  can  be  examined  (Ebeling,  2005).  The  COTS  based  component 
replacements  considered  for  this  modification  contract  consist  of  black  box  type  LRUs  that 
require  shorter  replacement  time  and  provide  ease  of  access.  For  this  reason,  Fleadquarters 
AMC  with  the  consultation  of  industry  experts  have  set  a  goal  of  a  repair  time  reduction  to  not 
more  than  30  minutes  on  average  for  modifications  related  repairs  compared  to  the  historical 
value  of  3  hours  from  equation  4.3. 

Threshold  Availability  at  the  30  minute  recommended  repair  time  would  be: 


=- 


.3257 


.3257  +  .0833  2  +  .0046 


=  .7944 


(4.8) 
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where  2  =  1/.5  hours. 


Objective  Availability  at  the  30  minute  recommended  repair  time  would  be: 


A  = 


.3257 


.3257 +  .0833  2 +  .0030 


=  .7951 


(4.9) 


As  seen  from  these  calculations,  maintainability  design  goals  present  even  further 
availability  improvements  beyond  the  reliability  improvements  of  78.52%  and  78.90% 
experienced  from  calculations  presented  in  equation  4.6  and  4.7.  The  visibility  of  the 
availability  gains  achieved  from  this  modification  program  would  be  valuable  for  any 
developer  tasked  with  setting  readiness  requirements  that  are  realistic  and  achievable. 


Table  7-  Aircraft  Availability  Improvement  Values 


3.07  Hr  MTTR 

0.5  Hr  MTTR 

Baseline  Availability 

.7791 

N/A 

Threshold  Availability 

.7852 

.7941 

Objective  Availability 

.7890 

.7951 

Resulting  Minimum  Number  of  Spares  Require 


Am  =  percentage  of  aircraft  operational  = 


40  arcft 
X 


=  0.7944 


(4.10) 


Solving  for  X: 


X  = 


40arcft 

0.7791 


=  51  after  rounding  up 


The  number  of  spares  =  51-  40.  To  keep  40  aircraft  operational  on  average  for  1  year, 
1 1  spares  are  required  at  both  threshold  and  objective  availability  values. 
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The  key  purpose  for  this  research  was  not  to  show  whether  these  increases  would 
provide  sizeable  gains,  or  optimize  reliability  to  cost  trade-offs,  but  to  develop  an  objective, 
standard,  repeatable  method  for  quantifying  the  Materiel  Availability  KPP  and  the  associated 
Materiel  Reliability  KSA  value  for  defense  weapon  system  requirements  documents. 
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V.  Recommendations 


Answering  the  Research  Question 

Air  Mobility  Command  (AMC)  lacks  a  standardized  method  to  establish  these 
mandatory  threshold  and  objective  values  for  the  Materiel  Availability  KPP  and  Materiel 
Reliability  KSA  for  modifications  to  existing  weapons  systems.  A  draft  guide  is  proposed  by 
the  Joint  Staff  with  the  purpose  of  assisting  program  decision  makers,  but  it’s  applicability  to 
the  AMC  problem  is  unclear. 

Is  the  draft  Reliability,  Availability,  Maintainability  and  Cost  Guide  (RAM-C)  that  the 
Office  of  the  Secretary  of  Defense  in  Collaboration  with  the  Joint  Staff prepared  applicable 
for  use  on  modification  program? 

Yes  this  draft  guide  is  applicable  for  modification  programs.  We  feel  that  better  results 
can  be  achieved  when  utilizing  this  draft  guide  if  recommendations  to  the  5  investigative 
questions  are  implemented. 

Investigative  Questions 

Question  1:  What  portions  of  the  guide  are  applicable  to  this  study  and  what  portions 
do  not  apply? 

The  primary  purpose  of  this  study  was  to  detennine  if  the  principles  and  processes 
contained  within  a  draft  RAM-C  guide  would  provide  assistance  in  helping  the  combat 
developer  in  establishing  reasonable,  balanced  mandatory  sustainment  requirements.  The 
concepts  and  principles  were  applied  to  an  ongoing  major  aircraft  modification  program  for 
planners  at  Headquarters  AMC. 

Unlike  new  weapons  systems  acquisitions,  most  modifications  are  undertaken  to 
improve  on  existing  capabilities  without  changing  the  basic  theory  of  employment  of  the 
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upgraded  system.  The  processes  outlining  the  development  of  the  CONOPS,  OMS/MP,  and 
FD/SC  do  not  seem  applicable  to  this  study  mainly  because  these  areas  would  remain 
unchanged  from  the  legacy  system.  Also,  the  infonnation  pertaining  to  maintenance  concepts 
and  product  support  were  of  little  use  for  this  modification,  considering  the  maintenance  and 
support  framework  would  be  unchanged  from  the  legacy  system. 

The  concepts  in  this  document  that  was  most  useful  was  the  infonnation  pertaining  to 
the  calculation  of  the  baseline  reliability  and  availability  metrics.  Also  the  spares  requirements 
calculation  was  very  helpful  and  insightful. 

Question  2:  What  modification  program  would  be  a  viable  candidate  for  the  use  in 
this  study? 

For  this  study  we  considered  two  in-progress  aircraft  modification  programs  for 
Headquarters  AMC,  the  C-  5  RERP  and  the  KC-135  CNS/ATM.  Both  programs  were 
modifications  to  aircraft  that  have  been  in  the  Air  Force  inventory  for  decades.  This  provided 
us  access  to  a  wealth  knowledge  and  historical  trend  data  that  is  required  to  assess  what 
information  that  is  pertinent  in  establishing  sustainment  requirements. 

In  choosing  our  aircraft  for  this  study  we  examined  issues  such  as  size  of  modification, 
accessibility  to  performance  data  and  phase  of  contract  acquisition.  Each  program  presented  a 
different  level  of  complexity  based  on  the  number  of  subsystems  to  be  upgraded.  Both 
programs  involved  some  form  of  COTS  systems  components  and  access  to  industry’s  estimated 
performance  data  was  available.  The  main  deciding  factor  between  the  two  modification 
programs  was  the  size  of  the  system  upgrade.  For  this  reason  the  KC-135  CNS/ATM  program 
was  chosen  as  the  program  for  study.  But  overall,  based  on  the  results  achieved  from  this 
research,  it  is  highly  conceivable  that  a  modification  as  complicated  as  the  C-5  RERP  could  be 
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accomplished  utilizing  the  same  principles  contained  within  this  study  given  an  ample  amount 
of  time  to  do  so. 

Question  3:  If  the  guide  is  deemed  applicable  for  use  in  modification  or  upgrade 
programs  what  areas  within  the  guide  requires  changes  and  what  changes  are 
recommended? 

The  concepts  presented  in  the  draft  guide  are  mainly  focused  on  acquisition  of  an 
entirely  new  weapon  system.  To  get  the  best  results  from  this  document  as  a  tool  for 
modification  programs  some  changes  should  be  made.  One  area  that  did  not  receive  much 
attention  was  historical  data  examination  and  sensitivity  analysis.  A  critical  insight  gained 
during  this  research  was  how  examination  of  different  factors  such  as  characteristic  life  values 
and  the  impact  such  evaluation  has  on  achieving  predicted  measures.  More  emphasis  should  be 
placed  on  reliability  sensitivity  analysis  and  availability  sensitivity  analysis.  The  document  as 
it  stands  now  only  examines  a  comparative  evaluation  of  competing  systems  without  evaluating 
the  variables  of  each  system  to  determine  how  this  analysis  affects  feasibility  and  overall 
availability. 

Because  the  document  does  not  address  the  historical  performance  of  weapons  system, 
the  creators  only  evaluate  sustainment  requirements  based  on  a  constant  failure  time 
distribution.  This  document  should  address  time  dependent  failure  distributions  based  on 
actual  failure  data. 

Question  4:  How  should  historical  reliability  and  maintenance  data  information 
systems  be  utilized  to  establish  availability  and  reliability  estimates? 

The  REMIS  maintenance  and  supply  data  system  that  was  utilized  in  this  research 
provided  a  tool  to  access  the  historical  failure  and  repair  time  information  needed  to  establish 
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sustainment  requirements.  Due  to  limitations  with  current  operating  time  reporting,  REMIS 
should  be  utilized  with  the  direct  assistance  of  subject  knowledge  experts  such  as  Headquarters 
AMC,  the  Tanker  program  office,  and  operational  KC-135  units  to  assist  in  the  validation  of 
any  assumptions  made  to  data  collection  and  analysis.  Also,  given  that  each  historical  data 
system  present  tools  not  share  between  systems,  time  should  be  allotted  to  cross  reference  each 
system  for  accuracy  and  to  fill  in  holes  that  can  give  a  better  picture  of  the  entire  program  to 
include  logistics  areas. 

Question  5:  Does  the  current  body  of  literature  concerning  Performance  Based 
Logistics  support  our  efforts  to  establish  sustainment  performance  parameters? 

Through  a  review  of  a  substantial  body  of  literature  outlining  the  theoretical  basis  for 
Performance  Base  Logistics  (PBL),  we  have  been  able  to  determine  that  PBL  can  be  used  as  a 
tool  to  aid  in  the  design  of  product  support  strategies  for  new  programs  or  major  modifications, 
or  as  we  reengineer  product  support  strategies  for  legacy  weapon  systems.  We  have  found  that 
it  is  commonly  perceived  that  perfonnance  based  logistics  is  likely  to  improve  product 
availability,  reliability,  maintainability  and  supportability  at  a  lower  cost  through  leveraging 
commercial  best  practices.  Typically  the  government  does  a  poor  job  at  optimizing 
sustainment  design  for  individual  weapons  system.  The  costs  effective  sustainment 
management  principles  that  commercial  industry  is  force  to  adhere  to  under  PBL  contracts  can 
provide  a  “best  practices”  acquisition  knowledge  road-map.  The  lessons  learned  from 
commercial  PBL  contract  implementations  can  be  utilized  to  reduced  the  Total  Life  Cycle 
Costs  and  reduce  the  logistics  footprint  requirement  under  traditional  acquisition  contracts. 
Assumptions 

In  order  to  conduct  this  research  some  key  assumptions  were  made: 
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1.  All  subsystems  identified  for  modification  in  KC-135  CNS/ATM  program  are  required 
for  flight  in-accordance  with  the  Minimum  Equipment  Safety  Listing  (MESL)  for  MDS 
KC135  as  of  December  01,  2008.  A  critical  failure  reported  on  any  one  of  these  system 
causing  that  system  to  be  non-mission  capable  causes  a  non-mission  capable  status  for 
the  aircraft  system  as  a  whole. 

2.  Failure  and  Repair  data  reported  in  REMIS  regarding  equipment  status  and  performance 
of  maintenance  actions  are  considered  accurate. 

3.  Immediately  upon  a  report  of  component  failure  repair  action  will  commence. 

4.  A  reported  failure  results  in  the  failed  subsystem’s  replacement  or  repair  to  a  level  of 
new  condition  (renewal  process). 

5.  The  reliability  measures  of  mean  time  between  failures  (MTBF)  will  be  an  equilibrium 
measure  based  on  the  evaluation  of  actual  failures  reported  over  the  12  month  reporting 
period  for  the  aircraft  and  units  of  study. 

6.  The  time  that  an  aircraft  is  available  for  flight  operation  is  based  on  the  time  from 
completion  of  a  maintenance  action  to  the  start  of  the  next  maintenance  actions  based 
on  a  24  period. 

7.  Information  received  from  Headquarters  Air  Mobility  Command  and  industry 
pertaining  to  Commercial  Off-the  Shelf  Technology  (COTS)  is  accurate. 

8.  Independence  between  subsystems  exist. 

Limitations 

One  of  the  main  requirements  in  detennining  time  between  successive  failures  is  the 
operating  time  of  the  equipment  being  evaluated.  One  of  the  limitations  and  a  known  system 
abnonnality  in  REMIS  is  that  the  system  does  not  output  current  operating  time  in  reports  that 
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contain  this  field.  The  method  of  using  a  24  hour  operating  clock  to  determine  MTBF  was 
validated  utilizing  data  obtained  from  Headquarters  Air  Mobility  Command.  Current  operating 
for  each  unit  under  evaluation  would  give  a  more  precise  measurement  for  each  unit  being 
evaluated. 

We  selected  only  the  Reliability  and  Maintainability  Infonnation  System  (REMIS)  will 
be  the  only  reliability  and  maintainability  infonnation  system  used  to  perform  this  research.  A 
more  thorough  evaluation  of  the  capabilities  and  shortcomings  of  each  data  source  could 
provide  an  improved  level  of  validation  of  the  legacy  systems  historical  performance 
measurements. 

Future  Research  Opportunities 

The  key  objective  of  this  research  was  not  to  achieve  optimized  materiel  readiness 
requirements  for  the  program  under  evaluation  in  this  study,  but  to  aid  in  detennining  if  the 
draft  guide  proposed  by  OSD  was  applicable  as  a  tool  to  in  establishing  mandatory  readiness 
requirements.  To  truly  gauge  the  effectiveness  of  this  guide  as  a  tool  to  establish  optimal 
readiness  goals,  costs  should  be  evaluated  in  addition  to  the  factors  address  in  this  research. 

The  sensitivity  analysis  performed  during  this  research  along  with  costs  data  should  provide 
better  insight  on  Total  Life  Cycle  Costs. 

Also,  an  analysis  of  a  program  that  is  much  more  complex  in  scope  and  number  of  the 
subsystems  or  components  to  be  modified  could  provide  better  insight  on  how  the  processes 
outlined  in  this  study  could  impact  overall  system  availability. 
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Appendix  1  -  PRP-4126  Screenshots 


USAF  -  PRP4126  -  On/Off  Equipment  Maintenance  Detail  Report 


User-Input  Select  Elements/Options  Were  As  Follows: 
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REMIS  Report  Is  For  Official  Use  Only  By  REMIS:  PCN  SG099-PRP4126 


Figure  10-  PRP-4126  On/Off  Equipment  Maintenance  Detail  Report 
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USAF  -  PRP41 26  -  On/Off  Equipment  Maintenance  Detail  Report 
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Figure  11-  On/Off  Equipment  Maintenance  Detail  Report 
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USAF  -  PRP41 26  -  Qn/Qff  Equipment  Maintenance  Detail  Report 
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Figure  12-  On/Off  Equipment  Maintenance  Detail  Report 
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Appendix  2  -  PRP-4126  Excel  Screenshots 
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Figure  13-  PRP-4126  On/Off  Equipment  Maintenance  Detail  Report  (Excel) 
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Appendix  3  -  Arena®  Input  Analyzer  Failure  Time  Distribution  Histograms 


Figure  14-  DFD  Failure  Time  Distribution 


Figure  15-  AOA  Failure  Time  Distribution 


Weibull,  K.  S.  p<0.01 

Figure  16-  AP  Failure  Time  Distribution 
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Weibull,  K.  S.  p<0.01 


Figure  17-  RA  Failure  Time  Distribution 


Exponential,  K.  S.  p  <  0.01 


Figure  18-  Aircraft  minus  Upgrades  Failure  Time  Distribution 
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Appendix  4  -  Arena®  Input  Analyzer  Repair  Time  Distribution  Histograms 


Figure  19-  DFD  Repair  Time  Distribution 


Figure  20-  AOA  Repair  Time  Distribution 
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Lognormal,  K.  S.  p  <  0.01 


JZH. 


Figure  22-  RA  Repair  Time  Distribution 


Figure  23-  Aircraft  minus  Upgrade  Repair  Time  Distribution 
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Appendix  5  -  Blue  Dart 


Master  Sergeant  Joe  M.  Blackman,  Student,  AFIT 
joe.blackman@afit.edu 

Word  Count:  731 

Quantification  of  Mandatory  Sustainment  Requirement 

To  emphasize  the  importance  of  sustainment,  the  DoD  Joint  Requirements  Oversight 
Council  addressed  sustained  Materiel  readiness  and  established  a  mandatory  Key  Performance 
Parameter  (KPP)  for  Materiel  Availability;  it  also  established  supporting  Key  System 
Attributes  (KSAs)  for  Materiel  Reliability  and  Ownership  Cost  (Chairman  of  the  Joint  Chiefs 
of  Staff  Manual  (CJCSM)  3170.01C,  2007).  KPPs  are  those  system  attributes  considered  most 
critical  or  essential  for  an  effective  military  capability.  KSAs  are  those  system  attributes 
considered  most  critical  or  essential  for  an  effective  military  capability  but  not  selected  as  a 
KPP.  KSAs  provide  decision  makers  with  an  additional  level  of  capability  prioritization  below 
the  KPP  but  with  senior  sponsor  leadership  control  (generally  4-star  level,  Defense  agency 
commander,  or  Principal  Staff  Assistant)  (Chairman  of  the  Joint  Chiefs  of  Staff  Manual 
(CJCSM)  3170. 01C,  2007). 

The  values  used  to  describe  each  KPP  and  KSA  are  defined  by  a  threshold  value  and  an 
objective  value.  The  threshold  value  for  an  attribute  is  the  minimum  acceptable  value 
considered  achievable  within  the  available  cost,  schedule,  and  technology  at  low-to-moderate 
risk.  Performance  below  the  threshold  value  is  not  operationally  effective  or  suitable.  The 
objective  value  for  an  attribute  is  the  desired  operational  goal  achievable  but  at  higher  risk  in 
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cost,  schedule,  and  technology.  Performance  above  the  objective  does  not  justify  additional 


expense. 

No  distinction  is  made  between  the  approaches  in  establishing  these  values  for  major 
system  acquisitions,  versus  smaller,  modification- focused  efforts  for  existing  systems.  In  some 
cases  a  range  of  stated  values  may  be  appropriate,  but  in  other  cases  the  best  approach  may  be 
to  specify  the  value  as  a  percentage  change  or  as  a  function  of  other  variables.  The  Joint  Staff 
proposed  guidance,  “Reliability,  Availability,  Maintainability  and  Cost  (RAM-C)  Guide,”  to 
assist  in  determining  these  values  for  major  acquisition  programs,  but  the  guidance  has  yet  to 
be  tested  on  modification  contracts.  To  assess  its  applicability,  we  performed  a  case  study  of  a 
recent  acquisition  program  under  consideration  by  Air  Mobility  Command.  We  sought  to 
apply  the  principles  put  forth  in  this  draft  guide  prepared  by  the  Office  of  the  Secretary  of 
Defense  (OSD)  in  Collaboration  with  the  Joint  Staff. 

Air  Mobility  Command  (AMC)  lacks  a  standardized  method  to  establish  these 
mandatory  threshold  and  objective  values  for  the  Materiel  Availability  KPP  and  Materiel 
Reliability  KSA  for  modifications  to  existing  weapons  systems.  In  order  to  address  this  issue 
the  draft  guide  proposed  by  the  Joint  Staff  was  applied  to  a  modification  effort  under 
consideration  by  AMC. 

The  Air  Force  Air  Mobility  Command’s  Directorate  of  Plans  and  Programs  approached 
us  with  a  request  for  help  in  quantifying  their  KPPs  for  several  ongoing  major  aircraft 
modifications,  and  will  support  our  research.  Therefore,  we  initially  focused  our  research  on 
recommending  a  standardized  method  for  quantifying  Materiel  readiness  sustainment  metrics 
for  Air  Force  Air  Mobility  Command’s  modification  efforts  but  we  designed  our  approach  and 
methods  to  be  applicable  to  all  major  system  modifications  and  upgrades. 
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To  achieve  this  objective,  we  used  an  ongoing  AMC  major  aircraft  modification  as  a 
case  study.  We  also  engaged  in  bi-lateral  discussions  with  program  and  subject  matter  experts, 
and  conducted  a  review  of  applicable  literature  to  develop  a  model  that  incorporates  multiple 
factors  impacting  each  sustaimnent  metrics.  This  model  was  based  on  the  concepts  and 
principles  put  forth  in  the  RAM-C  draft  guide.  The  decision  model  included  components  of 
predictive  modeling  and  sensitivity  analysis  of  the  subsystem  components  that  are  incorporated 
within  the  projected  upgrade  to  detennine  the  overall  affect  on  Materiel  readiness  values.  The 
sensitivity  analysis  performed  demonstrates  the  ability  of  the  procedures  utilized  in  the  study  to 
provide  decision  makers  with  additional  tools  that  provide  visibility  of  the  trade-offs  that  may 
be  capitalized  upon.  The  concepts  in  this  guide  would  allow  developers  to  meet  the  warfighters 
requirements  in  a  realistic  manner  while  simultaneously  minimizing  costs. 

The  processes  utilized  in  this  research  were  able  to  prove  that  the  draft  RAM-C  guide 
developed  by  the  Joint  Staff  is  a  valuable  tool  to  help  combat  developers  think  through  the  top- 
level  sustainment  requirements  for  RAM-C  early  in  the  requirements  generation  and  refinement 
phases  of  a  program.  With  the  addition  of  information  that  addresses  sensitivity  analysis 
required  and  the  historical  perfonnance  of  the  legacy  weapon  system  included  in  this  guide  we 
feel  that  this  document  could  prove  to  be  an  even  better  tool  when  applied  to  the  modification 
of  existing  weapons  systems. 

The  views  expressed  in  this  article  are  those  of  the  author  and  do  not  reflect  the  official 
policy  or  position  of  the  United  States  Air  Force,  Department  of  Defense,  or  the  US 
Government. 
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